Sending data back and forth
by on with No Comments

As you remember, last week I finished the new data model implementation within Field Day and downloading remote database data. Field Day writes sensor database to the local database respecting the new model — populating fieldday_streaming, fieldday_reading, and fieldday_spot.

The big hurdle that still needed to be hopped over was sending those newly populated tables back to the remote database. This week I got over that big hurdle. The user can now upload the local tables to the remote tables. Field Day gets all of the rows in the tables I mentioned above, starting with fieldday_spot because the other two have foreign keys that reference the spot table. Moving the rows from the reading and streaming table is simple, but moving the spot table has different characteristics. If people are pushing to the database at the same time, we have to make sure that we are only inserting one of each ‘trip, site, sector, and spot’ which make up the primary key in the database. Field Day takes the ‘tripID, siteID, sectorID, and spotID’ of each row in the local database and queries the remote database asking for the count where a line has those four columns that equal the values in the local table. If the count comes back zero, then Field Day inserts the row. Each tables updating is wrapped up in a transaction, so if something goes wrong Field Day rolls back the transaction and sets the database to it’s previous state. Once the inserting has successfully finished, the database in the local table is copied to external storage in an /archive directory and the tables are dropped and created again so they are empty.

I moved ‘Take a Sample’ on the main screen to ‘Sampling’ and created a new option under that menu for ‘Database Actions.’ The downloading from remote database activity was previously under the Settings option, but since the application is basically doing the same thing in each, I merged them in to one activity. There are three buttons for the user to choose from (which you can see in the picture below) — setup local database, upload tables, clear local database. Upon the press of any of the buttons, the user is asked to confirm.  When the user selects clear local database, just like after the user upload the tables, the local database is copies and wiped.

A couple other small things that I’ve worked on this week are integrating the Google Drive Android API and fixing small things with the sensor sampling activity. As for the Google Drive API, I’ve gotten Field Day to the point where it asks for permission to access Google Drive, but doesn’t do anything after that. To my knowledge, this API only allows the app to interact with files that were created by the application itself. I’m not 100% sure of that so I’m still working on it. Small things that are fixed are the user is notified if they haven’t set up their local database with a remote database, which means they can’t write anything to the local database. When a Spot number is put in the spot edit text on the sampling activity, Field Day checks to make sure that spotID doesn’t already exist in the local database if it does, then it doesn’t write to the database.

 

device-2016-05-19-094627

Light Blue Bean!
by on with No Comments

In my last post, I said that I was going to switch all of our individual fragments of sensors to one that is ‘Bluetooth Sensors.’ Since then, I have finished that and redesigned even more of Field Day.

After I finished switching all the fragments to one, I dove into working on BLE connection with the Light Blue Bean (LBB). Unfortunately, the LBB does not do Bluetooth the same way the Red Bear Labs shield does. The LBB uses something called ‘Scratch Characteristics.’ These are built-in 5 characteristics that can be written and read from the client (in this case, Field Day) and the server (the LBB). These characteristics have set UUIDs, so that means I can’t use the custom UUIDs I set for the RBL’s sensor. After browsing some of the android SDK code from Punchthrough (the people that made the LBB), I was able to determine the UUIDs that are used for the characteristics. Since Field Day is going to have to determine what kind of device it is connected to, I redesigned it so that the fragment is cleaner and doesn’t do any of that work. There are separate classes for a Bluetooth Sensor, GATT Client, and Bluetooth Service. The Service talks to the Sensor and the Sensor talks to the GATT client and determines what type of device it is connected to by checking the UUIDs. All the Fragment does now is say when it wants to write a message or read a message.

Punchthrough has sample code available for arduino and reading and writing the scratch characteristics. The examples along with the SDK has proven helpful when rewriting the android code to accommodate for the Bean. Unfortunately, not many people have used the Bean with Android. All of the examples are with iOS. Field Day is able to read scratch characteristics just fine from the Bean, but is currently unable to write to any characteristics. One huge problem with the bean is that everything is Bluetooth. It’s not like arduino where you plug in the device to your computer to upload code and power it on. This means that the Bean can only be connected to one thing at a time and thus, is really hard to debug. Typically when debugging arduino devices, I’m able to plug the device into my computer and watch the Serial monitor for debugging code I’ve added in. With the Bean, I cannot. It’s blind debugging and it sucks. I’m still working on figuring out how to write a characteristic and have the bean read that characteristic, but I think I’m getting closer. I hope to finish that this week.

This is getting really interesting
by on with 2 Comments

o On the archeology front, Rannveig found this article about the possibility of a new Viking site being excavated in Newfoundland. They must have stopped along the way from Scandinavia, and Skalanes seems like a nice place for that. Who knows what we’ll find.

o Nic has made some progress with LIDAR, although the gear is proving to be a bit delicate. We’ll probably order a second unit to work with. Here is an early image taken during today’s meeting. The unit is on the round table, on Nic’s screen is the developing point-cloud of the room and contents. If you look very closely you can see people on couches (/very/ closely…)

IMAG0390

o We need to sort a magnetometer before long, hopefully Patrick can loan us one.

o On the logistics front we need to make plane reservations fairly soon. All the lodging and transportation is sorted modulo the ferry to Grimsey, we’ll just wing that early in the morning of the day we travel.

o I started working with our Yocto altimeter recently, we’ll use this as part of the kit that provides more accurate x, y, and z geo-coordinates than consumer grade GPS chipsets alone do.

o Kristin and I worked through most of the details of the interface between FieldDay and the Arduino based sensor platforms. Here is a schematic of it, off to the left is the Postgres database where FieldDay pushes readings in CSV form.

IMAG0389

o I think it’s time we started the Iceland16 playlist, it looks like Spotify is currently the most popular platform for doing so.

 

BLE in Arduino and Android
by on with No Comments

I have accomplished quite a bit within the past week, if I do say so myself. Last week, I said that I had begun working on the BLE part of Field Day. Well, after some struggle I finished it! I am able to send a request to the Arduino device and upon receiving that request, the Arduino device sends a message back to the android device and writes to he screen (woo!). But the code is not pretty. Android provides a Bluetooth Low Service example in Android Studio, but it uses deprecated code. I researched and was able to modernize it.

Bluetooth Low Energy is pretty complicated as a service. It uses something called GATT (Generic Attribute Profile). The bluetooth server (arduino device) has a GATT profile and within that GATT profile there are services and services have characteristics. Each service can have multiple characteristics, but there can only be one TX and one RX (read and write) characteristic per service. I learned this the hard way. You can see a diagram of what I’m talking about below.

For the arduino side, we use Red Bear Labs BLE shield. There is a standard BLE library for the arduino but it’s really complex. I’ve read over the code multiple times and I’m just now grasping a little bit of it. RBL has constructed a wrapper for that code. They broke it down into 10 or so different functions, which I must say is mighty useful. I used that when constructing the Arduino code.

During my coding of the BLE on android I discovered that most of our Fragments of sensors are all the same except for the names of them. After talking with Charlie we’ve decided that we’re going to get rid of the individual fragments for each sensors and just have one that is ‘Bluetooth Sensors.’ I’m working on moving the BLE code to that setup now. Hopefully I will be done with it within the day.

 

GATT

Handlers handling handles
by on with 1 Comment

After some testing of the Handler implementation in Field Day, I’ve determined that it is actually working! I was finally able to look at the data that I had been writing to the database.

Databases in Android are stored in internal (private) memory. There are Android apps that let you look at the files on your device, but only for external storage. The internal storage is private. What I ended up doing was adding a function to write the database from internal storage to external storage. Once it was on external storage, I used a SQLite Viewer Android app to look at the database and it looks good!

I did some research into the application state when the device screen is turned off. In our old application, Seshat, the activities would die when the screen was off. That’s not good. We want to be able to turn off the screen and hang the Android device from our person some how. It’s useful to have our hands free. After a couple hours of research, I determined the libraries we want to look at are in the PowerManager library on Android, particularly the PARTIAL_WAKE_LOCK state. I wrap the methods I want to happen when the screen is off in what are essentially ‘start’ and ‘stop’ methods. The methods were not executing with the screen off, so I researched some more. I think what will ultimately happen is switching the SensorSampleActivity to a Service. A Service in Android is one of the top priority processes. It’s one of the last to die to if the device needs more memory. It can continue running even if the Activity the started it has died. Since the Activity is now working, I’m going to hold off in converting it to a Service.

The last thing I worked on this week was Bluetooth LE in Android. Tara gave me a BLE shield with an Arduino attached to test my code with those devices. Android provides a BLE example in Android Studio so I’ve been working with that. However, the example they provide has deprecated code so I’m working on modifying it to the new setup. It’s not working yet, but I’ve only been working on it for a couple of hours!

sqlite-fieldday
Sample of sqlite database on Field Day

I love the details…
by on with 1 Comment

The past week has been .23 zillion small things. A few travel related, a few data, and a lot of sensor platforms. The UAS and LIDAR gear is in-house, so far Nic has made the LIDAR spin (he said it was “working”…) and the UAS has charged batteries. I’ve been meeting with Anna and Katie looking at measuring the voltage potential generated by the pH testers. Anna and I worked-through the software architecture for connecting Arduino-based sensors to FieldDay via BlueTooth. There is some data aggregation to be done on the Arduino, and we should probably look at George’s libraries.

Tara and I built a rig out of Lego for spinning 50ML Falcon tubes, if the basic principle works we can fab it out of “real” materials. That was so much easier than the last thing she asked me for, a suction filtration device on the Sunday after Christmas in Leon, Nicaragua.

I worked with Deeksha and Eamon on the data organization, cleanup, visualization, etc. It’s moving along albeit a tad slowly.

The Lilly map sale is today, I’m going to see what I can find for topos of the Wayne county area that we can use for testing the orienteering gear. Kristin and I were interviewed by Mark Brim about how we collaborate, that was fun.

Databases, Threads, and aesthetics!
by on with No Comments

After getting most of the logistics sorted out, I’ve finally been able to get back to working on Field Day. Not a whole lot has been done because I spent most of my time working on logistics but I was able to get a few things done or started. The first thing I did was get rid of the black background for the Sensor Sample activity. We don’t have to keep it this color I was just tired of the black. I made it green because it reminded me of grass and being in the field. The next thing I added was EditTexts for Site, Sector, and Spot like Seshat had. In the data model, we also added a ‘Trip’ column but I didn’t think that there needed to be an option for that on the main screen. I put in option in the Settings page since it’s going to be constant throughout the entire trip. In Seshat, we had an option to select how often the stream to the database. I added that in Field Day as a Radio Group with customized buttons from 1 – 60 seconds. I haven’t done anything with those values yet but the visual and backbone is in place. (You can see these edits in the two pictures below — Sensor Sample on the left and Settings on the right). Ignore the ‘Table name for readings database’ on the Settings page for the time being because I have figured that out yet.

fielddaysettings

Not much to report…
by on with No Comments

I haven’t done much in the last week, to be honest. It was kind of a crazy week for me so I had to focus on some Admin things. I’m still looking at the best method for threads in Android and thinking about the Notepad/Lab Notebook/Checklist aspect of Field Day and the best methods and classes to implement it. If there’s anything that could improve the Notepad from Seshat. That uses a Navigation Drawer but to my knowledge, that has been deprecated in the new APIs, so more research is to be done.

Best method for threads in Android
by on with No Comments

The past week I worked on making the classes that pertain to sensor sampling platforms more generic. Previously, I had been working on just getting them working using the Built-In sensor sampling fragment. The Built-In fragment uses the built in android sensors on the hardware device which uses the Android Sensor class. The aSensor class and other similar classes that we are going to use for all sensor platforms (Database reading and writing, sensor sample fragment frame, and the list adapter) were using Android Sensor class methods. I worked on getting rid of those methods and replacing them with more generic methods that can be used for all sensor sampling types.

I have also been researching the best method for creating a thread in android classes. The SQLite work I’ve been doing is almost finished except it’s writing to the database too quickly which is making my primary keys fail. I am looking for best practices for writing to a database in a thread. In Seshat (our old application), we had a feature where the user could select how often they wanted to write to the database (in seconds). I plan to move this feature to Field Day but I want to research the best approach. In Seshat, we used a runOnUiThread method. This may be the best approach but there are other options to consider — Service, AsyncTask, Handler, Runnable, etc.

Back to Android and SQLite
by on with No Comments

Since the beginning of the semester, I’ve been working on Field Day again and somewhat sorting out git. At the end of last semester, we were having trouble with git and we’re still trying to figure that out. We’ve decided that we’re all going to stick on the same Android Studio version and upgrade at times we decide as a group. Nic has the most updated version 1.5, so we’re all going to stick to that.

The latest code push was code that broke my Field Day setup, so all of the code I’ve been working on hasn’t been pushed yet. I’ve started a new branch of our git repo called SQLite work that I’ve been committing my changes to. As a result, I’ve been learning a lot of git and branching and I think this may be the way to do so we don’t end up with broken code in our ‘master’ branch.

The part of Field Day I’ve been working on is the SQLite database. Previously, we wrote all of our readings to a CSV file but that gets quite messy. So, in this new application we’ve decided to go to SQLite. I’ve written a class called ReadingsDatabase which is a SQLiteOpenHelper class. This is used to create, delete, upgrade, update, query, etc the database. I’ve also written an interface that will be used to communicate between the Sensor Fragment classes and the main activity. I’ve been using the ‘Built-In Sensors’ fragment to test because we haven’t created any of the other sensors to test yet. I believe it is working, but the problem I’m running into is the class is writing to the database too quickly which is causing the primary keys I’ve set up to fail. I’m looking into a way to slow down the writing to the database. Once we figure out the git problem, I’m going to push my code.

1 2