It seems that everything is OK by now.
After some stressing weeks, the Christmas break seems to have a positive effect on everyone. New end dates for functional mapping have been defined and thus, according with the plan, we're on time again.
Tuesday, December 28, 2010
Thursday, December 23, 2010
Day 632 - Christmas Break
This is the last work day before a short Christmas break.
It seems the "I'll tell you how to do your job" symptom is getting worst.
Now management, instead of managing, is, literally, querying the target database with SQL in AS/400 and is creating a fuss by asking and making assumptions about things that he finds awkward.
The problem is that he doesn't know the target system that well and is just getting on everyone's nerves because people has to stop working the data migration and waste time to get data right to answer him and showing how he was wrong.
I hope this short Christmas break helps everyone to calm down.
It seems the "I'll tell you how to do your job" symptom is getting worst.
Now management, instead of managing, is, literally, querying the target database with SQL in AS/400 and is creating a fuss by asking and making assumptions about things that he finds awkward.
The problem is that he doesn't know the target system that well and is just getting on everyone's nerves because people has to stop working the data migration and waste time to get data right to answer him and showing how he was wrong.
I hope this short Christmas break helps everyone to calm down.
Day 630 - Qualification Preliminary Report
The preliminary report indicates a very good success rate.
It has been presented to management, and administration, during the weekly meeting.
But even with good results, management is really panicking and it's showing it semi-publicly.
Now management is trying to tell everybody how to do their work, me included.
Management really told me how to do my work, technically, with database details and everything. When we started discussing the problem I realised pretty quickly that actually, he didn't knew what he was talking about. So, I've just agreed with him and let it go. The other way was to "beat him up" in front of his entire team showing that he didn't not even had a clue about how the process was set up, and he knew even less about our requirements and the technical solutions we had to set up to met them.
This kind of action is one more panic symptom, and it's not helping the team.
It has been presented to management, and administration, during the weekly meeting.
But even with good results, management is really panicking and it's showing it semi-publicly.
Now management is trying to tell everybody how to do their work, me included.
Management really told me how to do my work, technically, with database details and everything. When we started discussing the problem I realised pretty quickly that actually, he didn't knew what he was talking about. So, I've just agreed with him and let it go. The other way was to "beat him up" in front of his entire team showing that he didn't not even had a clue about how the process was set up, and he knew even less about our requirements and the technical solutions we had to set up to met them.
This kind of action is one more panic symptom, and it's not helping the team.
Day 629 - Last Qualification Day
As I predicted, account took ages and it's loading has been aborted at 21:50.
This means that not only account was only partially loaded but also that some other residual, but important, data didn't had the chance to be loaded.
Management seems to be unhappy, but my initial report data indicate a very good success rate, except for three insurance claims areas.
This means that not only account was only partially loaded but also that some other residual, but important, data didn't had the chance to be loaded.
Management seems to be unhappy, but my initial report data indicate a very good success rate, except for three insurance claims areas.
Day 628 - Second Qualification Weekend Day
Some insurance claims have been loaded but only one area had no errors, all the other areas had insignificant success rates.
The data migration is running late because the car insurance data loading had to be repeated yesterday and account will take ages to load. This means that the data migration will not be completed during the weekend.
The data migration is running late because the car insurance data loading had to be repeated yesterday and account will take ages to load. This means that the data migration will not be completed during the weekend.
Day 627 - First Qualification Weekend Day
The first weekend day will be mainly for insurance data loading.
All insurances have problems, including the car insurance data.
It seems to be am insignificant amount of errors, but I was not expecting any errors from data insurance.
All insurances have problems, including the car insurance data.
It seems to be am insignificant amount of errors, but I was not expecting any errors from data insurance.
Day 626 - Car Insurance Problems
The loading of the car insurance data has gone totally wrong.
An unidentified problem in GIS has messed up over 11 hours of AS/400 work.
When trying the exact same loading during the day it went just fine, it's one of these awkward problems that happens too many times during the GIS data loading and that no one ever knows the cause...
This will force this migration to be pushed, since car insurances will have to be loaded again during the weekend.
I will be following closely this migration during the weekend.
If some more problems arise I'll be the first one to know and I'll sound the alarms.
I have scheduled a set of checkpoints during the weekend, starting Saturday early and ending Sunday night.
An unidentified problem in GIS has messed up over 11 hours of AS/400 work.
When trying the exact same loading during the day it went just fine, it's one of these awkward problems that happens too many times during the GIS data loading and that no one ever knows the cause...
This will force this migration to be pushed, since car insurances will have to be loaded again during the weekend.
I will be following closely this migration during the weekend.
If some more problems arise I'll be the first one to know and I'll sound the alarms.
I have scheduled a set of checkpoints during the weekend, starting Saturday early and ending Sunday night.
Day 624 - Entity Loading in Qualification
Today we've started a migration for Qualification environment for testing purposes.
Entities have been loading during dawn and it all went well.
This will go on for the hole week and weekend. I've predicted that this will end Monday dawn.
Entities have been loading during dawn and it all went well.
This will go on for the hole week and weekend. I've predicted that this will end Monday dawn.
Day 623 - New Processes
As expected, the new controlling team as set up a new set of controlling procedures, or processes, or, as they actually are, bureaucracies.
I've been asked to help setting this processes for the GIS data loading step.
I'll be glad to help, since it will allow me to actually make this a useful thing.
I've been asked to help setting this processes for the GIS data loading step.
I'll be glad to help, since it will allow me to actually make this a useful thing.
Day 616 - No Commitment
In the weakly meeting management tried to commit the team with new dates for some task that probably will delay in the next few days.
But since the team is not sure that this delay will happen and, if it happens, how much time will it be, there was no answer whatsoever.
Management was not happy with this situation. Currently there's a lot of friction between management and the migration team that resulted from the way management has treated it's own team along the way, and specially since the migration is part of the critical path.
But since the team is not sure that this delay will happen and, if it happens, how much time will it be, there was no answer whatsoever.
Management was not happy with this situation. Currently there's a lot of friction between management and the migration team that resulted from the way management has treated it's own team along the way, and specially since the migration is part of the critical path.
Day 609 - New Controlling Team
There's a new team in the project. It's an outside management consultancy that will be responsible for the project control.
Things will be pretty much the same, but I presume some more bureaucracy will appear soon, again under the label of "control"...
Things will be pretty much the same, but I presume some more bureaucracy will appear soon, again under the label of "control"...
Day 598 - Management Pressure
Since the data migration project is part of the project critial path, there's a lot more pressure on the team.
Management is pressuring the team because management is starting panicking.
And, as usual, when management is in panic mode, there's an increase of bureaucracy hidden under the label of "control".
Management needs to control in order to know really fast if things start to slide again.
Now, the migration team has to do more reporting documents, witch is not very pleasant when people are already working 10-12 hours per day for some consecutive weeks.
Management is pressuring the team because management is starting panicking.
And, as usual, when management is in panic mode, there's an increase of bureaucracy hidden under the label of "control".
Management needs to control in order to know really fast if things start to slide again.
Now, the migration team has to do more reporting documents, witch is not very pleasant when people are already working 10-12 hours per day for some consecutive weeks.
Friday, November 19, 2010
Day 595 - Critical Path
The data migration is now officially in the critical path of the entire corporation data migration project.
This are very bad news. But actually are not unexpected news.
Only this week I consider the data migration stable in a way that allow us to code the transformation rules with low change requests.
In fact, looking back, I sense we are precisely in the same point were we were when we went on vacations, except that now we have the insurance claims ready.
I sense that this data project has been stalled for three months, and I also sense I'm not the only one.
This are very bad news. But actually are not unexpected news.
Only this week I consider the data migration stable in a way that allow us to code the transformation rules with low change requests.
In fact, looking back, I sense we are precisely in the same point were we were when we went on vacations, except that now we have the insurance claims ready.
I sense that this data project has been stalled for three months, and I also sense I'm not the only one.
Day 582 - Nervous Managament and Team Delusion
Management is getting more nervous every time.
Things are still not fully stable and insurance claims are delayed. This is still not a real problem, but if this continues it can jeopardise April's dead line.
Instead of helping the migration team, management react with the classical control move, forcing people to report more things. This will, obviously, increase the team effort and consume more time.
Many times management seems to blame its own team for the project delays, and most of those times it is unfair.
This actually results from political problems, since management has committed itself in a way that it can only look good by blaming its own team.
People are, obviously, very unhappy with this situation. I actually feel that the team is kind of "wrecked", they just don't care as they did before. In the beginning the team embraced this project in a very positive way. They always fought for the best interest of the corporation, but they have been unfairly blamed so many times by the management that they just "cracked". I sense that, unfortunately, management has destroyed one great team, one kind of team that one doesn't see every day nor everywhere.
Things are still not fully stable and insurance claims are delayed. This is still not a real problem, but if this continues it can jeopardise April's dead line.
Instead of helping the migration team, management react with the classical control move, forcing people to report more things. This will, obviously, increase the team effort and consume more time.
Many times management seems to blame its own team for the project delays, and most of those times it is unfair.
This actually results from political problems, since management has committed itself in a way that it can only look good by blaming its own team.
People are, obviously, very unhappy with this situation. I actually feel that the team is kind of "wrecked", they just don't care as they did before. In the beginning the team embraced this project in a very positive way. They always fought for the best interest of the corporation, but they have been unfairly blamed so many times by the management that they just "cracked". I sense that, unfortunately, management has destroyed one great team, one kind of team that one doesn't see every day nor everywhere.
Day 575 - Sample Loading
After overcoming the DB2 problems, the sample has been loaded into the quality system.
We had some errors, some we were already expecting, and overall management were satisfied with it because the integration tests can be performed.
We had some errors, some we were already expecting, and overall management were satisfied with it because the integration tests can be performed.
Day 573 - DB2 Crash
This weekend we are going to load a specific sample in the quality system for integration tests.
Unfortunately this task were almost compromised because our DB2 staging area database crashed real hard.
It took me around 4 hours to put everything working again, and I had to reconstruct the entire staging area responsible for holding the transformed data.
Unfortunately this task were almost compromised because our DB2 staging area database crashed real hard.
It took me around 4 hours to put everything working again, and I had to reconstruct the entire staging area responsible for holding the transformed data.
Day 563 - Tests and Validations
The 13.5% of automatic validation are half-way developed. We've met in order to check if everyone was checking the same things the same way in all the systems.
As expected, there were some differences which will easily be corrected.
Meanwhile, the tests performed by the test team are still half-speed, things are not as stable as they should be, which means the team does not have much work.
As expected, there were some differences which will easily be corrected.
Meanwhile, the tests performed by the test team are still half-speed, things are not as stable as they should be, which means the team does not have much work.
Day 558 - Mapping Review
Since there is a mismatch between what target system has now implemented and current mappings, the mappings have been fully reviewed.
This consolidation action will result in a much desired stability.
Insurance policies are still not as stable as they were before the vacations, and management is starting to get nervous about it.
This consolidation action will result in a much desired stability.
Insurance policies are still not as stable as they were before the vacations, and management is starting to get nervous about it.
Day 549 - GIS Frozen
It seems that GIS has now frozen its development.
I hope this will bring some stability to the mappings is particular and to all the data migration project in general.
I hope this will bring some stability to the mappings is particular and to all the data migration project in general.
Day 534 - New Team Reshape
The team will suffer a new reshape.
Professionally I had planed to start a new project with a new client when this data migration ended next month.
But since the dead line has been pushed to April next year, that will not be possible.
Currently I'm the technical leader of this data migration and there is no one else in the team, or among any other team involved, that can replace me with a low cost and zero impact in the project.
Since I cannot leave and had agreed a new start date for another client, I had to negotiate my involvement in both projects. This was actually easy to do and the result is a balance between my involvement in both projects.
I already had requested one more person to our team, and this situation has just speed up that process. The team will actually be reinforced with two people, half-time.
They are both knowledge about Data Fusion and data migrations, so their integration will be easy and fast.
Professionally I had planed to start a new project with a new client when this data migration ended next month.
But since the dead line has been pushed to April next year, that will not be possible.
Currently I'm the technical leader of this data migration and there is no one else in the team, or among any other team involved, that can replace me with a low cost and zero impact in the project.
Since I cannot leave and had agreed a new start date for another client, I had to negotiate my involvement in both projects. This was actually easy to do and the result is a balance between my involvement in both projects.
I already had requested one more person to our team, and this situation has just speed up that process. The team will actually be reinforced with two people, half-time.
They are both knowledge about Data Fusion and data migrations, so their integration will be easy and fast.
Day 528 - The Start of a New Era
The new plan is now in action but things are about the same.
Insurance policy had a backset. Thins that were stable, are now being rejected and mappings are now suffering many changes.
Stability has still not been achieved and GIS is still a moving target, and the migration team is struggling to keep up with the new plan in order to comply with it.
Tests still have not full started because it's still not possible to load data into GIS.
The start of this new era is fictional because everything is about the same as before.
Insurance policy had a backset. Thins that were stable, are now being rejected and mappings are now suffering many changes.
Stability has still not been achieved and GIS is still a moving target, and the migration team is struggling to keep up with the new plan in order to comply with it.
Tests still have not full started because it's still not possible to load data into GIS.
The start of this new era is fictional because everything is about the same as before.
Monday, September 6, 2010
Day 521 - Re-Plan
Today we have re-planned.
The new dead-line will be April 2nd.
This leaves a lot of time to develop and test things.
Unfortunately management is, in my opinion, doing a terrible mistake when it comes to testing. I've personally helped in the draft test document, where around 100 tests that covered the technical part of the data migration, things such as control counts and sums. I've officially stated, more than once, that the test document should be expanded with the business tests, which should be around the 100 test also. In short, the testing document should cover around 200 tests, minimum.
My professional opinion on this has been ignored, and management identified only 27 tests to be implemented. This is, obviously, insufficient. Only 13.5% of the tests that should be performed will not guarantee the data migration quality.
This is clearly a high risk, but management has been so under pressure due to the time delays that it is making the most old, common and newbie mistake of them all: cutting on quality.
This is entirely a client problem, since its business will have to live with the data as it has been migrated. Everybody knows how bad, dirty and erroneous data impacts a business, but management seems to be ignoring this.
It looks like time is not the only issue with this project. It seems that cost has slip 50% already. No wonder administration is now taking a closer look and paying extra attention to this project.
The new dead-line will be April 2nd.
This leaves a lot of time to develop and test things.
Unfortunately management is, in my opinion, doing a terrible mistake when it comes to testing. I've personally helped in the draft test document, where around 100 tests that covered the technical part of the data migration, things such as control counts and sums. I've officially stated, more than once, that the test document should be expanded with the business tests, which should be around the 100 test also. In short, the testing document should cover around 200 tests, minimum.
My professional opinion on this has been ignored, and management identified only 27 tests to be implemented. This is, obviously, insufficient. Only 13.5% of the tests that should be performed will not guarantee the data migration quality.
This is clearly a high risk, but management has been so under pressure due to the time delays that it is making the most old, common and newbie mistake of them all: cutting on quality.
This is entirely a client problem, since its business will have to live with the data as it has been migrated. Everybody knows how bad, dirty and erroneous data impacts a business, but management seems to be ignoring this.
It looks like time is not the only issue with this project. It seems that cost has slip 50% already. No wonder administration is now taking a closer look and paying extra attention to this project.
Day 517 - Back from Vacations
We've just came back from our vacations and we've just found out that things have taken a twist.
The GIS is still not ready, instead of speeding up it actually delayed, thus it should be ready by the end of September.
Also, as I though, administration gave management one last chance to make it. It looks that this time there will be watch points, where management will decide to continue, or cancel the project, depending on its status.
We already have a working plan for this week, which kind of makes it serious when management says this time is for real.
The GIS is still not ready, instead of speeding up it actually delayed, thus it should be ready by the end of September.
Also, as I though, administration gave management one last chance to make it. It looks that this time there will be watch points, where management will decide to continue, or cancel the project, depending on its status.
We already have a working plan for this week, which kind of makes it serious when management says this time is for real.
Day 486 - First Simulation is No-Go
The first simulation will not happen this weekend.
As predicted, things must be re-planned since the dead lines will be impossible to accomplish mainly because the GIS software is not ready.
I've just discovered that the claims area is still under development and should be ready somewhere around mid August.
The re-plan will be a though decision for the administration, since management will have to explain very well why a new re-plan is required.
I have the idea that management has one last shot on getting this project done. I believe administration will allow this, last, re-plan, and that it will be the last one.
The data migration team will now go on vacation for the entire month of August.
We expect things to be better when we get back.
As predicted, things must be re-planned since the dead lines will be impossible to accomplish mainly because the GIS software is not ready.
I've just discovered that the claims area is still under development and should be ready somewhere around mid August.
The re-plan will be a though decision for the administration, since management will have to explain very well why a new re-plan is required.
I have the idea that management has one last shot on getting this project done. I believe administration will allow this, last, re-plan, and that it will be the last one.
The data migration team will now go on vacation for the entire month of August.
We expect things to be better when we get back.
Day 482 - Really Bad News
The tests over the weekend when somewhat OK, but the first simulation of the data migration is compromised.
Things are still pretty unstable and there isn't a single test for validation defined. The first simulation will receive a no-go decision from me, and probably from management to, which means a re-plan will most probably occur.
Things are still pretty unstable and there isn't a single test for validation defined. The first simulation will receive a no-go decision from me, and probably from management to, which means a re-plan will most probably occur.
Day 479 - Another Data Set for Weekend Tests
We have set up a new data set for the weekend tests, but things are so delayed that the minimum conditions for the first simulation to happen.
In particular, there isn't a single test defined to validate the data migration.
In particular, there isn't a single test defined to validate the data migration.
Monday, July 19, 2010
Day 475 - New GIS Version Ready
The new GIS version has been installed but the data we've prepared has not been loaded.
A new request for the same data has already been made in order to load it over the next weekend.
Within two weeks from the, still planned, first simulation, the target system, GIS, is still unstable, the claims are not fully mapped, account is still untested and automatic tests are still on paper.
We are precisely were we where some weeks ago, when management showed that did believe on this plan.
A new request for the same data has already been made in order to load it over the next weekend.
Within two weeks from the, still planned, first simulation, the target system, GIS, is still unstable, the claims are not fully mapped, account is still untested and automatic tests are still on paper.
We are precisely were we where some weeks ago, when management showed that did believe on this plan.
Day 472 - Bad, Bad News.
We've got the data ready for loading, as requested by management, for this weekend test.
When I went to talk with management, I was surprised to know that probably the data will not be loaded because a new GIS version was going to be installed. And it was not only me that was surprised, management didn't new this was going to happen!
Definitely management is unable to manage this project, planed things do not happen when they should and unplanned things with high impact happen without management acknowledgment.
Because of this new GIS version installation this means we have been working the last two days for nothing and the testing team will have to perform the same old ad hoc tests again.
This was also the day that the administration received the bad news. We, the data migration team, have officially stated that the immaturity of GIS, the current state of mapping rules, and the nonexistence of automated tests have made the first simulation, still scheduled for the last weekend of this month, impractical.
We also officially stated that this has a direct impact on the overall project and a new plan is now required because the current one will not happen.
When I went to talk with management, I was surprised to know that probably the data will not be loaded because a new GIS version was going to be installed. And it was not only me that was surprised, management didn't new this was going to happen!
Definitely management is unable to manage this project, planed things do not happen when they should and unplanned things with high impact happen without management acknowledgment.
Because of this new GIS version installation this means we have been working the last two days for nothing and the testing team will have to perform the same old ad hoc tests again.
This was also the day that the administration received the bad news. We, the data migration team, have officially stated that the immaturity of GIS, the current state of mapping rules, and the nonexistence of automated tests have made the first simulation, still scheduled for the last weekend of this month, impractical.
We also officially stated that this has a direct impact on the overall project and a new plan is now required because the current one will not happen.
Day 470 - New Data Set for Weekend Tests
During the weekly team meeting I got a request from management to prepare a very specific data set to test during this weekend.
It is not hard do get it, but the quantity of work involved will fulfill us for a couple of days.
During the meeting I've told management that by now the implementation of the automatic tests should have been finished. But in fact they have not even started. Plus, management has decided to cut the number of tests to implement and perform. I've officially stated that, in my professional opinion, the original number of tests should be expanded and not cut, since there was still many business things left out untested. But management things the opposite and they rule, it's their project.
Again, as I've been doing for over the past weeks, I've told management that the first simulation, and the entire plan, will be compromised if no action is taken this week.
It is not hard do get it, but the quantity of work involved will fulfill us for a couple of days.
During the meeting I've told management that by now the implementation of the automatic tests should have been finished. But in fact they have not even started. Plus, management has decided to cut the number of tests to implement and perform. I've officially stated that, in my professional opinion, the original number of tests should be expanded and not cut, since there was still many business things left out untested. But management things the opposite and they rule, it's their project.
Again, as I've been doing for over the past weeks, I've told management that the first simulation, and the entire plan, will be compromised if no action is taken this week.
Day 468 - New Week, Old Problems
Data has been loaded into GIS with some errors. We already expected errors from the account area, that was not properly tested, and the claims are not even close to be totally defined.
On the top of that, insurances have suffered a review which will result on a set of changes and, of course, a new set of tests.
Only three weeks left for the first simulation, but things are pretty unstable. I forecast a no-go decision for that first simulation. And if things don't get straight, I forecast a re-plan of the entire data migration project.
On the top of that, insurances have suffered a review which will result on a set of changes and, of course, a new set of tests.
Only three weeks left for the first simulation, but things are pretty unstable. I forecast a no-go decision for that first simulation. And if things don't get straight, I forecast a re-plan of the entire data migration project.
Day 464 - Small Test Planed
Things are not working out as expected. The first data loaded into GIS was rejected due to inconsistencies between the mapping rules and the GIS products configuration.
This is a recurring problem, every time a clean data load is performed GIS is not correctly configured to take it and errors that should not exist arise.
Account cannot be tested as expected because we need to prepare a small set of data to load into GIS this weekend. This has became the main focus for this week.
On the top of that, the automatic test strategy is still on paper. It is on paper for some weeks now and it seems it will not go anywhere anytime soon. The testing team is also tired of performing the same ad hoc tests again and again, they feel like little hamsters running on the wheel, they work hard but don't really go anywhere.
The first simulation is in less than 4 week and I'm already tired to tell management that account, accidents and tests are running late. If no action is performed, this will compromise the first simulation and, probably, the entire data migration project plan.
This is a recurring problem, every time a clean data load is performed GIS is not correctly configured to take it and errors that should not exist arise.
Account cannot be tested as expected because we need to prepare a small set of data to load into GIS this weekend. This has became the main focus for this week.
On the top of that, the automatic test strategy is still on paper. It is on paper for some weeks now and it seems it will not go anywhere anytime soon. The testing team is also tired of performing the same ad hoc tests again and again, they feel like little hamsters running on the wheel, they work hard but don't really go anywhere.
The first simulation is in less than 4 week and I'm already tired to tell management that account, accidents and tests are running late. If no action is performed, this will compromise the first simulation and, probably, the entire data migration project plan.
Day 461 - New GIS Version
The new GIS software version has been installed. I've just found out it actually should be ready three weeks ago. It seems the new version wasn't finished on time and that has cost the entire migration project 3 weeks of delay.
This means account has may be tested this week.
Some weeks ago management had planned to make up the lost time and put the project on track.
Unfortunately, reality is totally different, time is running out and the project is getting even more delayed every day...
This means account has may be tested this week.
Some weeks ago management had planned to make up the lost time and put the project on track.
Unfortunately, reality is totally different, time is running out and the project is getting even more delayed every day...
Day 454 - New GIS Version Required
A new GIS software version is required in order to overcome a problem in account that has been stalled for weeks.
Time is running out and the new GIS version should be available at the end of this week.
This will be a major improvement in the account area because the mapping and testing teams will finally be able to define it and test it.
Time is running out and the new GIS version should be available at the end of this week.
This will be a major improvement in the account area because the mapping and testing teams will finally be able to define it and test it.
Day 447 - Small Vacations
Lots of people, me included, has taken some time off due to a couple of holidays.
This means that the mapping team has not performed as fast as it was required, thus the project did not recover the lost time as it needed.
The actions taken by management and the mapping team show that it will be very difficult to comply with the current plan.
This means that the mapping team has not performed as fast as it was required, thus the project did not recover the lost time as it needed.
The actions taken by management and the mapping team show that it will be very difficult to comply with the current plan.
Day 427 - No Real Improvements
Things are not going as expected.
The mapping is still unstable and there are delays in solving the problems.
The test team is not working full time since there is not much to test.
Management is trying to act in order to put everything on track again, I sincerely hope it can go on track. There is still time, with an added effort, to make things work on time.
The mapping is still unstable and there are delays in solving the problems.
The test team is not working full time since there is not much to test.
Management is trying to act in order to put everything on track again, I sincerely hope it can go on track. There is still time, with an added effort, to make things work on time.
Friday, May 28, 2010
Day 422 - Unstable Mapping
Things are not going the way I, and the management team, was expecting.
After the restart, teams start working with focused on the current official project dates, even if they don't really believe it.
But the last GIS loadings we've performed showed that there was a major mapping regression. For instance car insurances were all loading correctly and now there isn't a single insurance that loads correctly, they have all been rejected.
It turns out that the products configuration in GIS, that should be concluded some weeks ago, has changed again.
And that was not all, all the product insurance mapping rules are changing. This mapping instability means that the project has an overall regression when there's only 8 weeks left to the first data migration simulation.
But this is not all. When the testing team tried to perform tests, they canceled the task because there was nothing that they could test. The migrated data that was loaded and available for testing will change soon.
After the restart, teams start working with focused on the current official project dates, even if they don't really believe it.
But the last GIS loadings we've performed showed that there was a major mapping regression. For instance car insurances were all loading correctly and now there isn't a single insurance that loads correctly, they have all been rejected.
It turns out that the products configuration in GIS, that should be concluded some weeks ago, has changed again.
And that was not all, all the product insurance mapping rules are changing. This mapping instability means that the project has an overall regression when there's only 8 weeks left to the first data migration simulation.
But this is not all. When the testing team tried to perform tests, they canceled the task because there was nothing that they could test. The migrated data that was loaded and available for testing will change soon.
Thursday, May 20, 2010
Day 413 - Restart
Today there was a migration meeting between the migration team, one management team responsible and I. It seems the migration will be performed in one shot and October is still the official date. But the management team is ready to take a four week delay, which means the migration would happen in November 1st.
This dates have been largely refuted by the migration team for several reasons. The most important of all is that they do not believe that the GIS implementation will be ready on time form compliance with these dates. This disbelieve has resulted in a very interesting discussion between management and the migration team.
The migration team is tired, the project has now an overall of two working years where these people have a work overload, they have to perform all their usual tasks plus they have to develop the migration. The worst problem came from the fact that historically the GIS implementation has failed to deliver the implementation of the defined requirements on time. Plus, there is still an huge amount of work to be done in this area, so the migration team is more than skeptical, they actually do not believe on these dates.
Worst, they are mad about the fact that the management team is being permissive with the GIS development delays and the features are not fully implementing to the define requirements.
After some pressure by the migration team, the management team admitted that they believe on these dates but do not put aside the possibility of another postpone.
On the top of all that, I've pointed out that there's still a considerable amount of work to be done regarding the tests and quality assurance. We already have identified counting checks and summary checks, but the identification of business rules validation is still work to be done.
Once the validations have been identified, they must be development on critical points: at data extraction, at data loading into the staging area, at data extraction from the staging area, and at data loading into GIS.
Nevertheless, the entire team has agreed to make another extra effort in order to try to comply with the current defined date. After a four week stop, the project restart is planed for tomorrow.
This dates have been largely refuted by the migration team for several reasons. The most important of all is that they do not believe that the GIS implementation will be ready on time form compliance with these dates. This disbelieve has resulted in a very interesting discussion between management and the migration team.
The migration team is tired, the project has now an overall of two working years where these people have a work overload, they have to perform all their usual tasks plus they have to develop the migration. The worst problem came from the fact that historically the GIS implementation has failed to deliver the implementation of the defined requirements on time. Plus, there is still an huge amount of work to be done in this area, so the migration team is more than skeptical, they actually do not believe on these dates.
Worst, they are mad about the fact that the management team is being permissive with the GIS development delays and the features are not fully implementing to the define requirements.
After some pressure by the migration team, the management team admitted that they believe on these dates but do not put aside the possibility of another postpone.
On the top of all that, I've pointed out that there's still a considerable amount of work to be done regarding the tests and quality assurance. We already have identified counting checks and summary checks, but the identification of business rules validation is still work to be done.
Once the validations have been identified, they must be development on critical points: at data extraction, at data loading into the staging area, at data extraction from the staging area, and at data loading into GIS.
Nevertheless, the entire team has agreed to make another extra effort in order to try to comply with the current defined date. After a four week stop, the project restart is planed for tomorrow.
Wednesday, May 5, 2010
Day 399 - Incremental Strategy Analysis
I've finished the requested analysis on performing the data migration incrementally.
Actually, it has a low impact for us because the requested split per product is already implemented and used on most of the data migration mappers. With less than three days work one can adapt the remaining mappers to use this strategy.
Actually, it has a low impact for us because the requested split per product is already implemented and used on most of the data migration mappers. With less than three days work one can adapt the remaining mappers to use this strategy.
Day 398 - Kickoff Planing of Migration Strategy
Management seems to believe that the current data migration will happen in the first weekend of October. We have met and discuss a lot about what needs to be done so that all the required tasks can be officially planed.
But the big news is not that management seems to believe on October as the dead-line. The big news is that management actually does not believe on that date as the final date. Management request me to evaluate the impact of an incremental data migration by product. They've said that this idea was fresh new, but they seem to believe that in October only half of the products will be ready on GIS and the rest will not be available for at least two more months. This new strategy will imply that one data migration will occur in October and another one will probably occur three months later, around January 2011.
But the big news is not that management seems to believe on October as the dead-line. The big news is that management actually does not believe on that date as the final date. Management request me to evaluate the impact of an incremental data migration by product. They've said that this idea was fresh new, but they seem to believe that in October only half of the products will be ready on GIS and the rest will not be available for at least two more months. This new strategy will imply that one data migration will occur in October and another one will probably occur three months later, around January 2011.
Day 391 - Team Leading
I'm now leading the data migration team.
In practice this means that now I decide, instead of discussing things with the ex-team leader, and I have to make all the status reports and other management stuff.
In practice this means that now I decide, instead of discussing things with the ex-team leader, and I have to make all the status reports and other management stuff.
Thursday, April 22, 2010
Day 386 - Official Plan Released
The new official data migration dates have been released.
It seems we're going to perform a simulation in the weekend before we all go to holidays, which will be the entire month of August. I don't really believe that this will happen since there is a lot of work to be done by the project sponsor teams, specially when it comes to quality. There was no more tests performed since the testing team has come form Easter holidays and it seems that there is no implementation plan regarding the tests we've identified some weeks ago.
We are still 3 months away, so it can actually be done if the sponsor acts fast.
When we get back from the holidays, in the first week of September, we will only be performing simulations until the real data migration happens in the beginning of October.
There are rumors from some sponsor people that the this new data migration date will not happen and that the project will be postponed again to January. These people are the same that have already predicted the other postpones and their justifications for it were always right. I'm afraid they are right again since they are pointing out the quality of the data migration and the quality of new system as major show stoppers. As I've stated before, a lot of testing is still required.
It seems we're going to perform a simulation in the weekend before we all go to holidays, which will be the entire month of August. I don't really believe that this will happen since there is a lot of work to be done by the project sponsor teams, specially when it comes to quality. There was no more tests performed since the testing team has come form Easter holidays and it seems that there is no implementation plan regarding the tests we've identified some weeks ago.
We are still 3 months away, so it can actually be done if the sponsor acts fast.
When we get back from the holidays, in the first week of September, we will only be performing simulations until the real data migration happens in the beginning of October.
There are rumors from some sponsor people that the this new data migration date will not happen and that the project will be postponed again to January. These people are the same that have already predicted the other postpones and their justifications for it were always right. I'm afraid they are right again since they are pointing out the quality of the data migration and the quality of new system as major show stoppers. As I've stated before, a lot of testing is still required.
Day 379 - Data Migration Team Leader is Leaving
Our data migration technical team leader is leaving. He had already planed to leave by the end of this data migration project, when it was scheduled for July. The fact that the project has been postponed didn't changed his plans, so he resigned.
This means that the team has shrink to two people and I will be assuming the technical team leader role. Possibly the team will be increased later on, since the data migration simulations require at least 3 people.
This means that the team has shrink to two people and I will be assuming the technical team leader role. Possibly the team will be increased later on, since the data migration simulations require at least 3 people.
Thursday, April 8, 2010
Day 372 - Project Due Date Postponed Again
The project due date has been postponed four more months. It looks likes the data migration will now happen in the 4th of October.
This happened mainly because the target system will not be fully developed up to June.
It looks management wants to keep the data migration plan to June and give us some months of force holidays. I do hope management won't go with this plan because it is impractical.
It is wrong to believe that the transformation rules can be frozen for 3 months when there will be massive development on the target system. And the project sponsor teams already know it by experience, whenever it was required to change something on the target system there was always a considerable amount of mapping rules that had to be redefined, implemented and tested. Plus, it is also wrong to believe that whatever tests the testing team will perform over this 3 months will not require any mapping changes.
This postpone is a great opportunity to define and implement the test plan. Some of the people from the testing team, which I've been talking to, have seen the number of preliminary tests we've identified and agree that it would be unfeasible to define and implement a testing plan if the data migration was to be performed in June.
This is actually the first time I'm on a data migration project were, once set, the official D-Day had been postponed. And in this project, it has already happened twice!
Like in the first postpone, and as far as I know, the data migration itself is one of the few projects that was on schedule by the previous plan.
As in the previous postpone, this probably means that we will have extra resources for most of the time, considering the amount of work of the data migration team in this stage.
But it is still not clear what management will do regarding all the changes that this postpone brings.
This happened mainly because the target system will not be fully developed up to June.
It looks management wants to keep the data migration plan to June and give us some months of force holidays. I do hope management won't go with this plan because it is impractical.
It is wrong to believe that the transformation rules can be frozen for 3 months when there will be massive development on the target system. And the project sponsor teams already know it by experience, whenever it was required to change something on the target system there was always a considerable amount of mapping rules that had to be redefined, implemented and tested. Plus, it is also wrong to believe that whatever tests the testing team will perform over this 3 months will not require any mapping changes.
This postpone is a great opportunity to define and implement the test plan. Some of the people from the testing team, which I've been talking to, have seen the number of preliminary tests we've identified and agree that it would be unfeasible to define and implement a testing plan if the data migration was to be performed in June.
This is actually the first time I'm on a data migration project were, once set, the official D-Day had been postponed. And in this project, it has already happened twice!
Like in the first postpone, and as far as I know, the data migration itself is one of the few projects that was on schedule by the previous plan.
As in the previous postpone, this probably means that we will have extra resources for most of the time, considering the amount of work of the data migration team in this stage.
But it is still not clear what management will do regarding all the changes that this postpone brings.
Wednesday, April 7, 2010
Day 371 - Tests Needed
The massive data migration performed over the weekend was positive.
We got some errors, some due to dependencies and but nothing critical nor that we were not expecting.
One positive result from this weekend action was a request from the management team. They loved the counts so much that they want us to identify some data tests that should be performed.
We've identified almost 100 simple data tests, most of them queries, that should be performed over the source data, the migrated data and the loaded data to check if there were no records lost nor information mismatch.
They should finalize the document with identical tests related with the business itself.
And it looks like the testing team will start gathering and testing again this week.
Still, we are worried with the tests and the testing plan because the tests are totally ad hoc and there is no testing plan whatsoever.
We got some errors, some due to dependencies and but nothing critical nor that we were not expecting.
One positive result from this weekend action was a request from the management team. They loved the counts so much that they want us to identify some data tests that should be performed.
We've identified almost 100 simple data tests, most of them queries, that should be performed over the source data, the migrated data and the loaded data to check if there were no records lost nor information mismatch.
They should finalize the document with identical tests related with the business itself.
And it looks like the testing team will start gathering and testing again this week.
Still, we are worried with the tests and the testing plan because the tests are totally ad hoc and there is no testing plan whatsoever.
Day 366 - Massive Data Migration
This next weekend will be bigger because of the Easter holidays and we will make a massive data migration and loading during this next three days. It will serve to test the performance and massive loading on a clean environment.
We have already prepared the files and have scheduled their loading in GIS during the next days.
We have performed simple counts in order to check if there were no records being lost among the way. The project sponsor management team loved it.
We have already prepared the files and have scheduled their loading in GIS during the next days.
We have performed simple counts in order to check if there were no records being lost among the way. The project sponsor management team loved it.
Day 365 - One Year
The data migration project has completed one operational year.
This is also the last day of one of our team members which was responsible for the GIS file loading and reporting.
The project is kind of stalled. There is no real advances of the data migration problems nor on the testing area. There is a critical error, a true show stopper, on the account area for over two weeks now, and it seems the only ones that are interested in solving it are the ones that can't solve it because they have no knowledge about it.
The tests continue to be an head ache because now the tests are stopped for the Easter holidays.
We are only a few weeks from the first simulation and there are still no tests formalized what so ever. By now, the tests should already be implemented and automated, but they don't even exist!
The project sponsor management team seems comfortable with this situation, but this is a serious problem and a critical concern for the data migration team.
This is also the last day of one of our team members which was responsible for the GIS file loading and reporting.
The project is kind of stalled. There is no real advances of the data migration problems nor on the testing area. There is a critical error, a true show stopper, on the account area for over two weeks now, and it seems the only ones that are interested in solving it are the ones that can't solve it because they have no knowledge about it.
The tests continue to be an head ache because now the tests are stopped for the Easter holidays.
We are only a few weeks from the first simulation and there are still no tests formalized what so ever. By now, the tests should already be implemented and automated, but they don't even exist!
The project sponsor management team seems comfortable with this situation, but this is a serious problem and a critical concern for the data migration team.
Monday, March 29, 2010
Day 363 - First Production Problems
The testing team was right. They needed more time to test the system.
The first problem found regards product integration and interface testing.
It looks that the data migrated by us is using a key that differs from a satellite system. We checked the data migrated and it complies with the specification, the mapping rules defined and the transformation rules implemented. This was also confirmed various times over the last 4 months by the tests performed on the migrated data.
The other system is sending a different key from the one we're sending.
This is an obvious integration error which would easily be found and corrected if integration tests had been performed.
With less than 3 month from the real data migration, this is not a good sign. Specially because we've already alerted several times to the need of consistent and automatic tests, something that no one here seems to care for except the testing team which, by the way, does not have the power to enforce it.
The first problem found regards product integration and interface testing.
It looks that the data migrated by us is using a key that differs from a satellite system. We checked the data migrated and it complies with the specification, the mapping rules defined and the transformation rules implemented. This was also confirmed various times over the last 4 months by the tests performed on the migrated data.
The other system is sending a different key from the one we're sending.
This is an obvious integration error which would easily be found and corrected if integration tests had been performed.
With less than 3 month from the real data migration, this is not a good sign. Specially because we've already alerted several times to the need of consistent and automatic tests, something that no one here seems to care for except the testing team which, by the way, does not have the power to enforce it.
Day 360 - First Product Go Live
Things didn't went exactly as planned.
Just minutes before the loading of the migrated data someone just noticed that there was something wrong with the configuration and a critical action took place.
This meant a delay of over 4 hours regarding the original plan but eventually everything went well and the first product in GIS is ready for the go live next Monday.
Just minutes before the loading of the migrated data someone just noticed that there was something wrong with the configuration and a critical action took place.
This meant a delay of over 4 hours regarding the original plan but eventually everything went well and the first product in GIS is ready for the go live next Monday.
Friday, March 26, 2010
Day 359 - Pre-Production Stress
People are getting pre-production stress.
Things should go smooth, but since people do know that the tests should have been better, they get stressed.
Today, one of the people responsible for the data migrated for the system go live tomorrow has just came to ask me why the data has only 379 records. That is the same person that defined the data source scope, the transformation rules and has been validating the this data for the last 6 months...
In the end, it was all correct, it was just a pre-production stress attack.
I think this shows how uncomfortable people really are with the data migration quality assurance procedure.
Things should go smooth, but since people do know that the tests should have been better, they get stressed.
Today, one of the people responsible for the data migrated for the system go live tomorrow has just came to ask me why the data has only 379 records. That is the same person that defined the data source scope, the transformation rules and has been validating the this data for the last 6 months...
In the end, it was all correct, it was just a pre-production stress attack.
I think this shows how uncomfortable people really are with the data migration quality assurance procedure.
Thursday, March 25, 2010
Day 358 - Go Live Files Ready
We have received the source files and have loaded them into our staging area.
We have executed the data migration for the entities and created the GIS loading files required for the go live.
The files have been loaded in the quality acceptance environment without a single problem. Tomorrow the testing team will check if everything is in order and, if so, the data will be loaded on the day after.
We have executed the data migration for the entities and created the GIS loading files required for the go live.
The files have been loaded in the quality acceptance environment without a single problem. Tomorrow the testing team will check if everything is in order and, if so, the data will be loaded on the day after.
Day 352 - Data Migration Plan For Go Live
We have just received the plan for the data migration part that is required for the new system kick-off.
We will receive the source data, loaded it in our staging area, execute the transformation rules for the entities and create the GIS loading files for the acceptance and production environments.
Since a postpone has not been granted for the new system go live date, the testing team seems to be working a bit harder and things seem to be a lot better.
We will receive the source data, loaded it in our staging area, execute the transformation rules for the entities and create the GIS loading files for the acceptance and production environments.
Since a postpone has not been granted for the new system go live date, the testing team seems to be working a bit harder and things seem to be a lot better.
Day 345 - New System Go Live
The new system will go live with a new product.
Since this is a new product, only a tiny part of the data included in the migration scope will be loaded at that time.
This will happen on 29th March.
There has been problems with the tests, which can resumed to the fact that the tests are not yelling the expected results and are seen as insufficient by many people involved in the process.
Management did tried to postpone the go live for two weeks. But the project sponsor has denied this request, so, the system will go live as planed.
Since this is a new product, only a tiny part of the data included in the migration scope will be loaded at that time.
This will happen on 29th March.
There has been problems with the tests, which can resumed to the fact that the tests are not yelling the expected results and are seen as insufficient by many people involved in the process.
Management did tried to postpone the go live for two weeks. But the project sponsor has denied this request, so, the system will go live as planed.
Wednesday, March 17, 2010
Day 335 - Reprioritization
The project sponsor is not very comfortable with the current dead line compliance, so management has reprioritized the migration of the business areas.
This happened because management believes that there is need for a greater focus on some business areas where it comes to the data migration.
In practice, this means that credit insurance has been defined as non-priority data to be migrated.
I'm actively involved in this business area and my opinion is that this is actually a mistake.
Credit insurance is equivalent with all other business areas when it comes to mapping and errors, and the other business areas are not "on hold" because of the migration.
The reason the project sponsor is afraid is because the mapping team cannot solve the problems faster or there are dependencies from other players and the testing team is no longer working daily.
There are several management mistakes here. Testing should be a priority now, but the tests have been reduced and the testing team, which was performing ad-hoc tests 4 hours per day, is only doing up to 16 hours of testing per week. Some tests should have been automated by now, but there is not even a plan to do so, so testing will continue to be ad-hoc.
Management should focus on forcing third party players to comply with their tasks and dates instead of reducing the scope of data being migrated. As an example, we are waiting for some data files for at least 6 weeks, but the third party responsible for it doesn't actually care. They've sent a couple of files last week, to shut us up, and they came all wrong...
This happened because management believes that there is need for a greater focus on some business areas where it comes to the data migration.
In practice, this means that credit insurance has been defined as non-priority data to be migrated.
I'm actively involved in this business area and my opinion is that this is actually a mistake.
Credit insurance is equivalent with all other business areas when it comes to mapping and errors, and the other business areas are not "on hold" because of the migration.
The reason the project sponsor is afraid is because the mapping team cannot solve the problems faster or there are dependencies from other players and the testing team is no longer working daily.
There are several management mistakes here. Testing should be a priority now, but the tests have been reduced and the testing team, which was performing ad-hoc tests 4 hours per day, is only doing up to 16 hours of testing per week. Some tests should have been automated by now, but there is not even a plan to do so, so testing will continue to be ad-hoc.
Management should focus on forcing third party players to comply with their tasks and dates instead of reducing the scope of data being migrated. As an example, we are waiting for some data files for at least 6 weeks, but the third party responsible for it doesn't actually care. They've sent a couple of files last week, to shut us up, and they came all wrong...
Day 332 - Yet, Another Team Reshape
There has been yet another team reshape.
In practice this means that there is currently only two business people responsible for the mappings, one which aims to more practical actions and another that aims for business rules validation.
In practice this means that there is currently only two business people responsible for the mappings, one which aims to more practical actions and another that aims for business rules validation.
Friday, February 19, 2010
Day 325 - Sponsor Team Reshape
The team is constantly changing from the beginning of the project and that is not helping the project at all.
This time, the sponsor team has suffered another reshape and one key person will move away from the data migration project.
The person how's being transferred has been reassigned to another task, but this person is one of the few key team members because he knows all the the areas like no one else. The person that is replacing him only knows a small subset of what he knows.
I believe the sponsor team has made a bad move regarding the data migration project.
This time, the sponsor team has suffered another reshape and one key person will move away from the data migration project.
The person how's being transferred has been reassigned to another task, but this person is one of the few key team members because he knows all the the areas like no one else. The person that is replacing him only knows a small subset of what he knows.
I believe the sponsor team has made a bad move regarding the data migration project.
Monday, February 15, 2010
Day 317 - Team Reshape
Since the project has been rescheduled, it was necessary to reshape our team.
This is necessary because our work load has not changed, it was just the project dead line that has been postponed.
The result of such reshape is still not clear, but the team will be downsized from 4 to 3 people. This does not exclude a temporary downsize to 2 people for up to one month within a couple of months or so. Only time will tell.
This is necessary because our work load has not changed, it was just the project dead line that has been postponed.
The result of such reshape is still not clear, but the team will be downsized from 4 to 3 people. This does not exclude a temporary downsize to 2 people for up to one month within a couple of months or so. Only time will tell.
Day 314 - Waiting for Resolutions
We have solved most of the issues reported, mostly were wrong mapping specifications, but some implementations bugs were also found.
Almost all issues are set for the functional team to solve and relate with rule specification.
Things are at a stable speed, but a bit slow from my point of view.
Almost all issues are set for the functional team to solve and relate with rule specification.
Things are at a stable speed, but a bit slow from my point of view.
Monday, February 1, 2010
Day 307 - Good Migration Results
The prioritization had some real effects.
Some things that were stalled for weeks were finally resolved and this weekend migration was the best we ever had. Nevertheless, there's still some very old issues, 3 months old, that need to be fixed.
GIS has also been cleaned up, which means the loading was performed without any trash on the system, and the test team can now perform the tests without calling us to check if there's a real problem or is it just some dirty data false problem.
Things look promising from now on. Let's see if it keeps this way.
Some things that were stalled for weeks were finally resolved and this weekend migration was the best we ever had. Nevertheless, there's still some very old issues, 3 months old, that need to be fixed.
GIS has also been cleaned up, which means the loading was performed without any trash on the system, and the test team can now perform the tests without calling us to check if there's a real problem or is it just some dirty data false problem.
Things look promising from now on. Let's see if it keeps this way.
Wednesday, January 27, 2010
Day 301 - Priorities
Finally management has shaken things up and is enforcing the teams to close issues, specially issues that are open for too long.
This is part of a bigger strategy, finalizing the mapping and loading of some business areas. There have been three areas selected, clients, simulations and credit insurance, I'm involved in two of them and I'm happy to see that some of these things are now moving into cruse speed.
This is part of a bigger strategy, finalizing the mapping and loading of some business areas. There have been three areas selected, clients, simulations and credit insurance, I'm involved in two of them and I'm happy to see that some of these things are now moving into cruse speed.
Friday, January 22, 2010
297 - Data Loading Files Performance Problem
The migration scope has increased.
New source data has come into play and thus new mapping specifications have been made.
This has resulted in a really big data loading performance problem.
Our data transformation procedure is still fast, but the creation of the GIS loader files from the transformed data is starting to give us some headaches.
The GIS data loader has a flat file structure that is very verbose, the loading of each single value of each record is done through a 300 character text file line.
This means that a single database table row is transformed into, something like, the same number of text lines as the number of columns the row has. Plus the file and records header and footer structure.
As an example, house insurances data transformation is performed in around 4 hours, sequential time, and it generates about 184 millions of records. This is all performed in the transformation server.
These records are then exported into the GIS data loader file format from the Windows directly into the AS/400. This procedure is now taking much time, over 6 hours, sequentially, in the best case scenario.
This is obviously too much time, so we are exploring several hypotheses, ranging from creating a parallel file write process; writing the files locally, with and without compression, and transfer them via FTP to AS/400; cluster indexes with full coverage maintained in a different disk; splitting the database schemes across several disks.
Some of these techniques can, and will, be combined.
We have to tune the process from our side since there is not much the GIS or the AS/400 can do when it comes to massive data load tuning.
We are facing a lot of hard work in these next days.
New source data has come into play and thus new mapping specifications have been made.
This has resulted in a really big data loading performance problem.
Our data transformation procedure is still fast, but the creation of the GIS loader files from the transformed data is starting to give us some headaches.
The GIS data loader has a flat file structure that is very verbose, the loading of each single value of each record is done through a 300 character text file line.
This means that a single database table row is transformed into, something like, the same number of text lines as the number of columns the row has. Plus the file and records header and footer structure.
As an example, house insurances data transformation is performed in around 4 hours, sequential time, and it generates about 184 millions of records. This is all performed in the transformation server.
These records are then exported into the GIS data loader file format from the Windows directly into the AS/400. This procedure is now taking much time, over 6 hours, sequentially, in the best case scenario.
This is obviously too much time, so we are exploring several hypotheses, ranging from creating a parallel file write process; writing the files locally, with and without compression, and transfer them via FTP to AS/400; cluster indexes with full coverage maintained in a different disk; splitting the database schemes across several disks.
Some of these techniques can, and will, be combined.
We have to tune the process from our side since there is not much the GIS or the AS/400 can do when it comes to massive data load tuning.
We are facing a lot of hard work in these next days.
Day 293 - Loading Status Queries
It seems the loading procedure feedback queries cannot be optimized any further by us.
We don't even have full comprehension about the data model, since there is no documentation and we had to learn by direct observation and trail and error.
We request help to the GIS team so that they help us in validating the queries and tuning the process.
We don't even have full comprehension about the data model, since there is no documentation and we had to learn by direct observation and trail and error.
We request help to the GIS team so that they help us in validating the queries and tuning the process.
Wednesday, January 13, 2010
Day 287 - Project Due Date Postponed
The project due date has been postponed almost two months. The D-Day date is now June 1st.
This happened because sub-projects related with the target system, such as testing, were delaying.
This is actually the first time I'm on a data migration project were, once set, the official D-Day had been postponed.
As far as I know, the data migration itself was one of the few projects that was on schedule.
The data migration will have to comply with this date and this probably mean that for one month we will have one extra resource. This is still unclear and thus the resize of the team is still under analysis.
This happened because sub-projects related with the target system, such as testing, were delaying.
This is actually the first time I'm on a data migration project were, once set, the official D-Day had been postponed.
As far as I know, the data migration itself was one of the few projects that was on schedule.
The data migration will have to comply with this date and this probably mean that for one month we will have one extra resource. This is still unclear and thus the resize of the team is still under analysis.
Wednesday, January 6, 2010
Day 279 - Happy New Year and Happy News
Clients are now being loaded with a marginal number of rejections.
The GIS configuration issues still have an impact but that problem is being solved quickly.
I'm still around the report load query problem, but it is not easy to test the performance on a slow system, since it takes too much time to process a query and return its result.
The GIS configuration issues still have an impact but that problem is being solved quickly.
I'm still around the report load query problem, but it is not easy to test the performance on a slow system, since it takes too much time to process a query and return its result.
Day 273 - Configuration Issues
The GIS configuration is changing and that has a direct impact in the mapping rules.
We are now facing an increase number of data being rejected by GIS because its configuration has been changed but the mapping hasn't.
We are now facing an increase number of data being rejected by GIS because its configuration has been changed but the mapping hasn't.
Day 266 - Christmas Break
The project will be stopped for a short Christmas break.
The overall number of data being rejected by the GIS loader is decreasing every day.
But we are now facing a performance problem again. It takes too much time to query GIS in order to know how many errors and what time of errors occurred in the loading procedure. I'll have to take a look at these query performance so that the loading report does not take half-day for a simple auto claim data load.
The overall number of data being rejected by the GIS loader is decreasing every day.
But we are now facing a performance problem again. It takes too much time to query GIS in order to know how many errors and what time of errors occurred in the loading procedure. I'll have to take a look at these query performance so that the loading report does not take half-day for a simple auto claim data load.
Day 264 - Error Free Loading
Finally we are start to get error free data loading.
Some types of clients are already loading into GIS and this is reducing the number of claims rejected by the GIS loader.
I'm expecting to stop worrying with client problems soon.
Some types of clients are already loading into GIS and this is reducing the number of claims rejected by the GIS loader.
I'm expecting to stop worrying with client problems soon.
Day 260 - Problem Prioritization
The problem solving problem is finally being solved.
Management has prioritized the problems to be fixed and is now working faster in the problem assignment.
Management has prioritized the problems to be fixed and is now working faster in the problem assignment.
Day 253 - Problem Solving is a Problem
Some problems do take too much time to be solved, they get stacked in the sponsor queue. Sometimes management does not assigned the problems by the team members fast enough, sometimes the team depends on other people and sometimes the team simply does not have a quick answer.
This is a real problem since we are now facing a dependency problem.
There are unsolved problems in the client area which is keeping clients to be integrated in GIS. And since there are clients missing, many claims are being rejected.
And since there are claims rejected, account is also being rejected.
This requires a fast answer from the management in order to be solved.
This is a real problem since we are now facing a dependency problem.
There are unsolved problems in the client area which is keeping clients to be integrated in GIS. And since there are clients missing, many claims are being rejected.
And since there are claims rejected, account is also being rejected.
This requires a fast answer from the management in order to be solved.
Day 250 - Team Rearrangements
There is a team management small problem.
The teams seem to be rearranged from time to time and this instability is not good for the sake of the project.
For instance, one person that was responsible for the client data migration, has been reassigned several times to other areas, like account and claims, and is now in the testing team. Two other people started to be responsible for the client data migration, one for each system, but it seems that the new people depend on the initial person.
There is another person that is responsible for the data extraction for one of the source systems but it seems now that it is also part of the mapping team.
This happens less on other areas, but it seems that there is an unofficially set of people that is assigned to all areas.
There is a clear group help spirit, which is great, because this eliminates political problems as "this is my migration area, so keep out".
The teams seem to be rearranged from time to time and this instability is not good for the sake of the project.
For instance, one person that was responsible for the client data migration, has been reassigned several times to other areas, like account and claims, and is now in the testing team. Two other people started to be responsible for the client data migration, one for each system, but it seems that the new people depend on the initial person.
There is another person that is responsible for the data extraction for one of the source systems but it seems now that it is also part of the mapping team.
This happens less on other areas, but it seems that there is an unofficially set of people that is assigned to all areas.
There is a clear group help spirit, which is great, because this eliminates political problems as "this is my migration area, so keep out".
Day 240 - Cleanup Procedure Status
The AS/400 clean up procedure and tuning procedures has resulted in a better performance.
The performance increase was not substantial, but since the domain includes millions of records, any minor improvement results in an overall improvement.
The performance increase was not substantial, but since the domain includes millions of records, any minor improvement results in an overall improvement.
Day 245 - Testing team
Management has finally assigned a test team to working half-day every day.
This is taking the tests to cruse speed.
Every day there are problems detected, rules changed and minor changes that have a great impact on the migration. The number of rejected records by the GIS data loader is starting to decrease.
This is taking the tests to cruse speed.
Every day there are problems detected, rules changed and minor changes that have a great impact on the migration. The number of rejected records by the GIS data loader is starting to decrease.
Subscribe to:
Posts (Atom)
Posts
