Day 46 - Performance Tests

I know it's Saturday, but a lot of data migrations work is done during the weekends. Specially during the last weeks before the end of the project.

But this Saturday, we were focused on performance tests. We had the AS/400 entirely for us for about 3 hours.
We executed performance tests for the transformation rules engine and for the GIS data loader.

As excepted, the performance increased both with a single process and with multiple processes. for both systems.
In the transformation engine we've executed up to 6 parallel migrations, but the maximum gain was below that, the 4th or 5th, depending on the transformation features, more I/O bound or more CPU bound.

There was not time to perform one interesting test though, running the transformation engine at the same time as the GIS data loader.

Nevertheless, the outcome was what we expected: to have a dedicated Intel machine running Linux for the exclusive usage of the Data Fusion. This will allow GIS data loader to have more resources, since we're not competing.

Here's some nice charts from the tests performed.

Day 34 - Denial Phase

The meeting with the project sponsor went almost as expected.
They loved the idea of having a separate server to execute the transformation rules, but the AS/400 administrators are in denial.
AS/400 administrators don't want to believe that AS/400 is not that as fas as they all brag about. If I had spent as much money as an AS/400 costs, I would be denying those facts too... It's just too painful to hear that in certain tasks, the AS/400 can be beaten by a much cheaper 8 CPU Intel machine running Linux...

By the way, the AS/400 were we're testing, the development environment, has the following features:

• iSeries 570
• 1 CPU Power 5 at 1.65GHz
• 3000 CPW
• 10GB RAM DDR1
• Internal HD: 1.6TB

The AS/400 administrators now want us to perform tests on the qualification machine:

• System i Power 6 570
• 1 CPU a 4.7GHz (maximum of 4 processors)
• 5000 CPW (maximum of 20000 CPWs)
• 20GB RAM DDR2 400MHz (maximum of 90GB)
• External HD: 2.4TB

I bet the test will turn out the same results, since the transformation engine is CPU bound, but nevertheless, we'll do the test during a weekend where there's no activity on the machine.

./M6

Day 30 - Performace Tests

There's a set of performance tests to do before choosing the final migration infrastructure.

There's tests over the initial data extraction from OS/390 and loading into AS/400.

There's tests over the data transformation rules.

There's tests over the final data loading into GIS.

From this set of tests, only ours, the data transformation rules, have been performed so far.

As expected, AS/400 is not a fast machine to run Data Fusion transformation engine. We already knew this, but AS/400 system administrators always say we're wrong and that they have a fantastic and fast machine. Their pride has been hurt...

As an example, running entirely in AS/400, 400000 source records have migrated at ~800 records per secord. Note that this was a local data migration execution, the source and target databases are on the same machine as the transformation engine.

But running the same transformation engine on my, not very fast, desktop computer with 1GB of RAM, the same 400000 records migrated at ~1400 records per second. Note that on this test the source data was retrieved from AS/400, transformed on my desktop compute, and sent back to AS/400, i.e. the data had to travel over a common 100 Mbit network.

The protocol loading interface was also subject to tests and it's CPU bounded, We have one single processor, of a 4 processor machine, and it consumes 100% of that CPU. When running on AS/400, it perfomed ~12000 records per second. Running on the desktop it does only ~8000 records per second.

The scalability tests on AS/400, with a single CPU for us, proved that a linear execution is faster than a two parallel executions. We haven't had the chance to test parallel execution with more CPUs.

We've been discussing what was the best approach allow us to scale and one thing is for sure, we need to run the transformation engine outside the AS/400, because on the D day we have to share, as it's happening now, the machine with others, specifically with GIS data loading.

Probably we'll have a 6 or 8 CPU machine running Linux and connected to AS/400 over a 1 Gbit network. This will allow us to run multiple transformation engines and free AS/400 for the other data migration processes.