Building a Big Data

It is Friday again, a long – waited weekend is ahead of us!

It was fairly warm yesterday with temperature above freeze, but an unexpected heavy snow storm kicked in late afternoon.  Most snow melted on the ground, with sharply dropped temperature last night, the wet roads have been frozen into a huge “skating ground” by this morning.  The rush hour commute became a true nightmare.

I had option to work at home, but the remote work could make the tasks I planned to complete by the end of today impossible.  Thus I fought the fear of driving on very icy roads, struggled through packed traffic along the way, and finally made to the headquarter at downtown.

The tasks in my mind for today was to build a big data containing the detailed race distributions throughout the entire country.

The office area was almost empty since most people chose to work at home.  I turned on my computer and connected to bank’s main server system.  I first started downloading the raw files from one of the government websites.  After 53 zipped files were transferred onto my local drive and were unzipped, I picked two sub-files from each of 53 unzipped files.  Then I wrote a macro to convert 106 sub-files from flat into a desired format.

Here came the fun part, I stacked 53 of 106 datasets together to form a single dataset, and stacked other 53 together to form another single dataset.  The final step was to merge these two datasets to build a big data containing over 14 million records.

Each step in the processes of downloading, transferring, converting, stacking and merging took from a few minutes to over 20 minutes.  Instead of staring on the screen to wait for the results, I eased myself through some “side – tasks” as singing old Chinese songs as if nobody was listening, doing office yoga postures as if nobody was watching, looking at the snow sculptures through office window as if they were melting away in next minute.  The tenuous brain labor immediately became enjoyable activities throughout the workday.

Late Friday afternoon, I finished planned tasks and turned off my computer before walking out of the headquarter.  It was still cold and icy outside but I felt totally relieved.  From the time when I took over the project a few months ago, I have been striving to build something new from ground – zero.  After numerous researches, readings, phone calls and meetings, I finally created this big data which will play a critical role in the upcoming analytical projects.

Right now it is the time to relax, I was on my way for Friday evening yoga practice.


Here is a related post:

3 thoughts on “Building a Big Data

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s