Sending data back and forth

by Kristin Muterspaw on 2016-05-21 with No Comments

As you remember, last week I finished the new data model implementation within Field Day and downloading remote database data. Field Day writes sensor database to the local database respecting the new model — populating fieldday_streaming, fieldday_reading, and fieldday_spot.

The big hurdle that still needed to be hopped over was sending those newly populated tables back to the remote database. This week I got over that big hurdle. The user can now upload the local tables to the remote tables. Field Day gets all of the rows in the tables I mentioned above, starting with fieldday_spot because the other two have foreign keys that reference the spot table. Moving the rows from the reading and streaming table is simple, but moving the spot table has different characteristics. If people are pushing to the database at the same time, we have to make sure that we are only inserting one of each ‘trip, site, sector, and spot’ which make up the primary key in the database. Field Day takes the ‘tripID, siteID, sectorID, and spotID’ of each row in the local database and queries the remote database asking for the count where a line has those four columns that equal the values in the local table. If the count comes back zero, then Field Day inserts the row. Each tables updating is wrapped up in a transaction, so if something goes wrong Field Day rolls back the transaction and sets the database to it’s previous state. Once the inserting has successfully finished, the database in the local table is copied to external storage in an /archive directory and the tables are dropped and created again so they are empty.

I moved ‘Take a Sample’ on the main screen to ‘Sampling’ and created a new option under that menu for ‘Database Actions.’ The downloading from remote database activity was previously under the Settings option, but since the application is basically doing the same thing in each, I merged them in to one activity. There are three buttons for the user to choose from (which you can see in the picture below) — setup local database, upload tables, clear local database. Upon the press of any of the buttons, the user is asked to confirm. When the user selects clear local database, just like after the user upload the tables, the local database is copies and wiped.

A couple other small things that I’ve worked on this week are integrating the Google Drive Android API and fixing small things with the sensor sampling activity. As for the Google Drive API, I’ve gotten Field Day to the point where it asks for permission to access Google Drive, but doesn’t do anything after that. To my knowledge, this API only allows the app to interact with files that were created by the application itself. I’m not 100% sure of that so I’m still working on it. Small things that are fixed are the user is notified if they haven’t set up their local database with a remote database, which means they can’t write anything to the local database. When a Spot number is put in the spot edit text on the sampling activity, Field Day checks to make sure that spotID doesn’t already exist in the local database if it does, then it doesn’t write to the database.

Scheming…

by Charlie Peck on 2016-05-07 with 1 Comment

After a week or so of tinkering we have a new version of the database schema that supports Field Day and the visualization stack. Here is the fancy entity relationship diagram for it:

Field Day will load trip, site, and sector information from Postgres (config option) and cache them on the tablet. Deeksha is committed to moving the data from Iceland13 & 14 and Nicaragua14 into this new structure.

Our credit card looks like one of Dali’s clocks…

by Charlie Peck on 2016-04-25 with No Comments

Kristin booked our tickets (yes!), and I ordered high precision barometric pressure sensors (to be used for altimeters), Adafruit Feathers to use as the micro-controllers for the ambiance/orienteering and soil (field and bench) platforms. This model has BLE and a battery charging circuit. I built a simple rig for the NIR sensor and scissor table, this is one of the bench soil platforms.

The next thing I would like to wrestle is reconciling the data stream that comes through the sensors into the database. Given that we’ll be collecting readings from multiple sensor platforms at each spot we need a way to aggregate those. This mirrors what happens with 911 incident data, in that case it’s the call number that binds all the different responses (fire, EMS, etc.) to an incident together.

Lots to do, 58 days to do it in…

Let’s design some stuff

by Charlie Peck on 2016-04-18 with No Comments

Tara and I designed and prototyped a couple of bench and field devices recently. We have a first pass at a Falcon tube turner, a soil microbe quantifier and a soil temperature and humidity probe.

The Falcon tube turner is designed to be used on a bench to mix soil samples as part of the pH, etc. protocols. Currently it’s based on Lego components, if that basic design works we can easily fabricate a couple robust enough for our needs.

Tara has an idea to quantify the amount of soil microbes by measuring their CO2 respiration. We have data for this from 16S extractions done at one of archeology sites at Skalanes, we can re-sample there and compare the two methods. Pictured below is our first pass at a device for doing this.

The soil temperature and humidity probe is a combination of a stainless steel soil coring tool with the two probes built-in to the end of the tube, see below (kind-of).

I’ve been working on reconciling the data model with the implications that Kristin brought when we worked-through the sensor <-> Arduino <-> BLE <-> Android <-> data model. Stay-tuned for the details.

There’s only 28 thousand days…

by Charlie Peck on 2016-04-02 with No Comments

Actually it’s only about 80 days until we leave but Alicia Keys’ song is stuck in my head after hearing it this morning when we were working in the lab. I do know the answer to her second question, Iceland.

Eamon and Deeksha are making progress on the data, data model, and viz tool. We are going to create a function that measures the distance between two lat,lon pairs to make annotating the site information easier and to provide a hook for the bounding-box interface. We wire-framed the viz tool UI, Eamon is going to construct a message to Patrick about open map tiles that we can cache on our devices.

Erin is chugging along with QGIS, we owe Patrick an adult beverage for suggesting that we use it. Nic recently discovered that there is a plug-in for it that supports LIDAR generated point-clouds in a layer (as a layer?).

I ran into Kelly Gaither (chair of XSEDE16) at IU on Friday, she strongly encouraged us to produce a poster for XSEDE16 on the UAS + LIDAR + machine learning package. She thinks it’s quite unique. Unfortunately she did remember me from the conference last summer when I hid under the table during the awards ceremony.

Next up for me is working on the high resolution Z platform and the soil temperature platform. I would like to work on Field Day too.

I love the details…

by Charlie Peck on 2016-03-28 with 1 Comment

The past week has been .23 zillion small things. A few travel related, a few data, and a lot of sensor platforms. The UAS and LIDAR gear is in-house, so far Nic has made the LIDAR spin (he said it was “working”…) and the UAS has charged batteries. I’ve been meeting with Anna and Katie looking at measuring the voltage potential generated by the pH testers. Anna and I worked-through the software architecture for connecting Arduino-based sensors to FieldDay via BlueTooth. There is some data aggregation to be done on the Arduino, and we should probably look at George’s libraries.

Tara and I built a rig out of Lego for spinning 50ML Falcon tubes, if the basic principle works we can fab it out of “real” materials. That was so much easier than the last thing she asked me for, a suction filtration device on the Sunday after Christmas in Leon, Nicaragua.

I worked with Deeksha and Eamon on the data organization, cleanup, visualization, etc. It’s moving along albeit a tad slowly.

The Lilly map sale is today, I’m going to see what I can find for topos of the Wayne county area that we can use for testing the orienteering gear. Kristin and I were interviewed by Mark Brim about how we collaborate, that was fun.

Databases, Threads, and aesthetics!

by Kristin Muterspaw on 2016-03-07 with No Comments

After getting most of the logistics sorted out, I’ve finally been able to get back to working on Field Day. Not a whole lot has been done because I spent most of my time working on logistics but I was able to get a few things done or started. The first thing I did was get rid of the black background for the Sensor Sample activity. We don’t have to keep it this color I was just tired of the black. I made it green because it reminded me of grass and being in the field. The next thing I added was EditTexts for Site, Sector, and Spot like Seshat had. In the data model, we also added a ‘Trip’ column but I didn’t think that there needed to be an option for that on the main screen. I put in option in the Settings page since it’s going to be constant throughout the entire trip. In Seshat, we had an option to select how often the stream to the database. I added that in Field Day as a Radio Group with customized buttons from 1 – 60 seconds. I haven’t done anything with those values yet but the visual and backbone is in place. (You can see these edits in the two pictures below — Sensor Sample on the left and Settings on the right). Ignore the ‘Table name for readings database’ on the Settings page for the time being because I have figured that out yet.

SQLite DB for Data.

by Kristin Muterspaw on 2015-11-08 with 1 Comment

On Saturday, Charlie and I discussed the notion of persistent data for the Field Day android application. Previously, we would stream all of our data to a CSV — even empty lines or entries that are missing data in some spots, and this would cause data to be in a bunch of different data models.

I recently read a chapter on SQLite databases in Android and it seems really powerful. We’ve decided that we are going to use a SQLite database instead of just writing to a CSV file. With this, we could create primary keys and easily pick out all the lines that have empty fields for one column or another. Using a SQLite database will also be useful for our plans to implement an ‘On-Bench’ and ‘In-Field’ option for the soil platform. There are some sensor data that you wouldn’t take in the field and would need to do later that day — like pH or organic content. Using a SQLite database, we’ll be able to simply ask for the sample ID from earlier that day and update it with the pH and Organic Content. Our data will be less messy and we’ll have less random lines of data floating around.

Cleaning Up

by Deeksha Srinath on 2015-11-02 with No Comments

I’ve continued working on my master table.I’ve been doing two things to the data,cleaning up the format it’s in,and substituting dummy values for those records where we don’t have values.I’ve also been doing a reasonableness check with the data by comparing our calendars and where the calendars said we were to the data we have.This has let me know what data we can safely get rid of,like the couple of testing streams we have from around hostels.

I’m almost done with cleaning up and should be able to just put all of my nice clean data (all in one format!!) into the fieldscience db.

Database design tools

by Deeksha Srinath on 2015-09-21 with No Comments

I spent this past week learning about what options exist as far as small scale database diagram tools exist.
I found a lot of tools that are standalone desktop applications,but once I decided that we needed a design tool that was a browser based application,the search was narrowed down quite a bit.I think being able to collaboratively look at the current designs lends itself to the kind of work we’re trying to do,so this is the final list of what I’ve narrowed the search down to:

I’m partial to using Vertabelo at this point,having played around with it for a while.I really like the interface and it’s pretty intuitive.It’s also free as long as the project is small,and we definitely come under what Vertabelo identifies as such.