This past week, I worked on getting some database views ready that would show us the results for the sampling day. This is so each evening in Iceland, we’re able to effectively and quickly QA the data we have and check to make sure what we have matches with what we should have. This is now set up so that we have views of data from the last 24 hours for both streaming and readings tables in the database. The views are currently set up so we can see the recordtime, latitude, longitude, elevation, sensortype and sensor value for the streaming table. The readings table view displays the time, site, sector, spot, sensortype and value for the last 24 hours.

I also finished up the ER diagram for the data model as it now stands in our database with a couple of changes from the last iteration.

I began working on learning enough about flask this week to start adapting the viz tool to work with the current data model, and expect to make more headway on that this week. I also started writing up and documenting all our tables and why we chose to structure them the way we did, along with explanations for the relationships between the tables and how we chose those.

This past week, the new data model was implemented after some K,C and I talked about it for the past few weeks. We now have multiple tables in the model, with a table for basically every data point. I have been looking at Iceland 2014 data to figure out what data we can move over into this new model. I’ve learned that there isn’t very much data we’d like to move over from Iceland 2014, although almost all the Nicaragua data is good to move over (in the fall, once we’re back from the Iceland scrum.) The distance function came in really handy here, and made me really glad that we finally all have the old data in a single, easily searchable place. We’re not moving the data over into the new model, but it’s easy to query it where it is now (something that hasn’t been possible for a while)

So for now, we are keeping all the Iceland 2014 and Nicaragua data in the old model on Postgres, and focusing on this years’ data, and getting that working in time.

Since my last post was a a very long time ago, this is what has happened with the data clean-up over the past weeks:

Eamon and I both cleaned up all the data we could find individually. Since we’d worked separately, when we compared the cleaned data sets that we each had, they turned out to be different. Neither of us had all the data individually, but when our data sets were combined, the list was exhaustive of all permutations of Iceland 2014 and Nicaragua 2014 data. Then, with Charlie’s help, we were able to determine which of the data sets we needed to zorch. It turns out that a significant chunk of our data sets needed to be zorched, because we each had thousands of rows of testing data or data taken in the car while the group was driving.

After much too much time wrestling data in spreadsheets, the readings table is finally in the field science database where further clean-up relating to sectors and spots can be done. As of right now, most Iceland 2014 data has no sector or spot data. With Charlie’s help, I can now populate the sectors.This should be way quicker to do by date, time stamp and lat long coordinates now that we have it all in the database.

Next up, I will be working to create/adapt a function that measures the distance between a pair of lat long coordinates. This should help with further clean-up and with the bounding box interface.