Sending data back and forth

with No Comments

As you remember, last week I finished the new data model implementation within Field Day and downloading remote database data. Field Day writes sensor database to the local database respecting the new model — populating fieldday_streaming, fieldday_reading, and fieldday_spot.

The big hurdle that still needed to be hopped over was sending those newly populated tables back to the remote database. This week I got over that big hurdle. The user can now upload the local tables to the remote tables. Field Day gets all of the rows in the tables I mentioned above, starting with fieldday_spot because the other two have foreign keys that reference the spot table. Moving the rows from the reading and streaming table is simple, but moving the spot table has different characteristics. If people are pushing to the database at the same time, we have to make sure that we are only inserting one of each ‘trip, site, sector, and spot’ which make up the primary key in the database. Field Day takes the ‘tripID, siteID, sectorID, and spotID’ of each row in the local database and queries the remote database asking for the count where a line has those four columns that equal the values in the local table. If the count comes back zero, then Field Day inserts the row. Each tables updating is wrapped up in a transaction, so if something goes wrong Field Day rolls back the transaction and sets the database to it’s previous state. Once the inserting has successfully finished, the database in the local table is copied to external storage in an /archive directory and the tables are dropped and created again so they are empty.

I moved ‘Take a Sample’ on the main screen to ‘Sampling’ and created a new option under that menu for ‘Database Actions.’ The downloading from remote database activity was previously under the Settings option, but since the application is basically doing the same thing in each, I merged them in to one activity. There are three buttons for the user to choose from (which you can see in the picture below) — setup local database, upload tables, clear local database. Upon the press of any of the buttons, the user is asked to confirm.  When the user selects clear local database, just like after the user upload the tables, the local database is copies and wiped.

A couple other small things that I’ve worked on this week are integrating the Google Drive Android API and fixing small things with the sensor sampling activity. As for the Google Drive API, I’ve gotten Field Day to the point where it asks for permission to access Google Drive, but doesn’t do anything after that. To my knowledge, this API only allows the app to interact with files that were created by the application itself. I’m not 100% sure of that so I’m still working on it. Small things that are fixed are the user is notified if they haven’t set up their local database with a remote database, which means they can’t write anything to the local database. When a Spot number is put in the spot edit text on the sampling activity, Field Day checks to make sure that spotID doesn’t already exist in the local database if it does, then it doesn’t write to the database.

 

device-2016-05-19-094627

Let’s design some stuff

with No Comments

Tara and I designed and prototyped a couple of bench and field devices recently. We have a first pass at a Falcon tube turner, a soil microbe quantifier and a soil temperature and humidity probe.

The Falcon tube turner is designed to be used on a bench to mix soil samples as part of the pH, etc. protocols. Currently it’s based on Lego components, if that basic design works we can easily fabricate a couple robust enough for our needs.

Tara has an idea to quantify the amount of soil microbes by measuring their CO2 respiration. We have data for this from 16S extractions done at one of archeology sites at Skalanes, we can re-sample there and compare the two methods. Pictured below is our first pass at a device for doing this.

IMAG0398

The soil temperature and humidity probe is a combination of a stainless steel soil coring tool with the two probes built-in to the end of the tube, see below (kind-of).

IMAG0399

I’ve been working on reconciling the data model with the implications that Kristin brought when we worked-through the sensor <-> Arduino <-> BLE <-> Android <-> data model. Stay-tuned for the details.

Light Blue Bean!

with No Comments

In my last post, I said that I was going to switch all of our individual fragments of sensors to one that is ‘Bluetooth Sensors.’ Since then, I have finished that and redesigned even more of Field Day.

After I finished switching all the fragments to one, I dove into working on BLE connection with the Light Blue Bean (LBB). Unfortunately, the LBB does not do Bluetooth the same way the Red Bear Labs shield does. The LBB uses something called ‘Scratch Characteristics.’ These are built-in 5 characteristics that can be written and read from the client (in this case, Field Day) and the server (the LBB). These characteristics have set UUIDs, so that means I can’t use the custom UUIDs I set for the RBL’s sensor. After browsing some of the android SDK code from Punchthrough (the people that made the LBB), I was able to determine the UUIDs that are used for the characteristics. Since Field Day is going to have to determine what kind of device it is connected to, I redesigned it so that the fragment is cleaner and doesn’t do any of that work. There are separate classes for a Bluetooth Sensor, GATT Client, and Bluetooth Service. The Service talks to the Sensor and the Sensor talks to the GATT client and determines what type of device it is connected to by checking the UUIDs. All the Fragment does now is say when it wants to write a message or read a message.

Punchthrough has sample code available for arduino and reading and writing the scratch characteristics. The examples along with the SDK has proven helpful when rewriting the android code to accommodate for the Bean. Unfortunately, not many people have used the Bean with Android. All of the examples are with iOS. Field Day is able to read scratch characteristics just fine from the Bean, but is currently unable to write to any characteristics. One huge problem with the bean is that everything is Bluetooth. It’s not like arduino where you plug in the device to your computer to upload code and power it on. This means that the Bean can only be connected to one thing at a time and thus, is really hard to debug. Typically when debugging arduino devices, I’m able to plug the device into my computer and watch the Serial monitor for debugging code I’ve added in. With the Bean, I cannot. It’s blind debugging and it sucks. I’m still working on figuring out how to write a characteristic and have the bean read that characteristic, but I think I’m getting closer. I hope to finish that this week.

BLE in Arduino and Android

with No Comments

I have accomplished quite a bit within the past week, if I do say so myself. Last week, I said that I had begun working on the BLE part of Field Day. Well, after some struggle I finished it! I am able to send a request to the Arduino device and upon receiving that request, the Arduino device sends a message back to the android device and writes to he screen (woo!). But the code is not pretty. Android provides a Bluetooth Low Service example in Android Studio, but it uses deprecated code. I researched and was able to modernize it.

Bluetooth Low Energy is pretty complicated as a service. It uses something called GATT (Generic Attribute Profile). The bluetooth server (arduino device) has a GATT profile and within that GATT profile there are services and services have characteristics. Each service can have multiple characteristics, but there can only be one TX and one RX (read and write) characteristic per service. I learned this the hard way. You can see a diagram of what I’m talking about below.

For the arduino side, we use Red Bear Labs BLE shield. There is a standard BLE library for the arduino but it’s really complex. I’ve read over the code multiple times and I’m just now grasping a little bit of it. RBL has constructed a wrapper for that code. They broke it down into 10 or so different functions, which I must say is mighty useful. I used that when constructing the Arduino code.

During my coding of the BLE on android I discovered that most of our Fragments of sensors are all the same except for the names of them. After talking with Charlie we’ve decided that we’re going to get rid of the individual fragments for each sensors and just have one that is ‘Bluetooth Sensors.’ I’m working on moving the BLE code to that setup now. Hopefully I will be done with it within the day.

 

GATT

Handlers handling handles

with 1 Comment

After some testing of the Handler implementation in Field Day, I’ve determined that it is actually working! I was finally able to look at the data that I had been writing to the database.

Databases in Android are stored in internal (private) memory. There are Android apps that let you look at the files on your device, but only for external storage. The internal storage is private. What I ended up doing was adding a function to write the database from internal storage to external storage. Once it was on external storage, I used a SQLite Viewer Android app to look at the database and it looks good!

I did some research into the application state when the device screen is turned off. In our old application, Seshat, the activities would die when the screen was off. That’s not good. We want to be able to turn off the screen and hang the Android device from our person some how. It’s useful to have our hands free. After a couple hours of research, I determined the libraries we want to look at are in the PowerManager library on Android, particularly the PARTIAL_WAKE_LOCK state. I wrap the methods I want to happen when the screen is off in what are essentially ‘start’ and ‘stop’ methods. The methods were not executing with the screen off, so I researched some more. I think what will ultimately happen is switching the SensorSampleActivity to a Service. A Service in Android is one of the top priority processes. It’s one of the last to die to if the device needs more memory. It can continue running even if the Activity the started it has died. Since the Activity is now working, I’m going to hold off in converting it to a Service.

The last thing I worked on this week was Bluetooth LE in Android. Tara gave me a BLE shield with an Arduino attached to test my code with those devices. Android provides a BLE example in Android Studio so I’ve been working with that. However, the example they provide has deprecated code so I’m working on modifying it to the new setup. It’s not working yet, but I’ve only been working on it for a couple of hours!

sqlite-fieldday
Sample of sqlite database on Field Day

Databases, Threads, and aesthetics!

with No Comments

After getting most of the logistics sorted out, I’ve finally been able to get back to working on Field Day. Not a whole lot has been done because I spent most of my time working on logistics but I was able to get a few things done or started. The first thing I did was get rid of the black background for the Sensor Sample activity. We don’t have to keep it this color I was just tired of the black. I made it green because it reminded me of grass and being in the field. The next thing I added was EditTexts for Site, Sector, and Spot like Seshat had. In the data model, we also added a ‘Trip’ column but I didn’t think that there needed to be an option for that on the main screen. I put in option in the Settings page since it’s going to be constant throughout the entire trip. In Seshat, we had an option to select how often the stream to the database. I added that in Field Day as a Radio Group with customized buttons from 1 – 60 seconds. I haven’t done anything with those values yet but the visual and backbone is in place. (You can see these edits in the two pictures below — Sensor Sample on the left and Settings on the right). Ignore the ‘Table name for readings database’ on the Settings page for the time being because I have figured that out yet.

fielddaysettings

Not much to report…

with No Comments

I haven’t done much in the last week, to be honest. It was kind of a crazy week for me so I had to focus on some Admin things. I’m still looking at the best method for threads in Android and thinking about the Notepad/Lab Notebook/Checklist aspect of Field Day and the best methods and classes to implement it. If there’s anything that could improve the Notepad from Seshat. That uses a Navigation Drawer but to my knowledge, that has been deprecated in the new APIs, so more research is to be done.

Best method for threads in Android

with No Comments

The past week I worked on making the classes that pertain to sensor sampling platforms more generic. Previously, I had been working on just getting them working using the Built-In sensor sampling fragment. The Built-In fragment uses the built in android sensors on the hardware device which uses the Android Sensor class. The aSensor class and other similar classes that we are going to use for all sensor platforms (Database reading and writing, sensor sample fragment frame, and the list adapter) were using Android Sensor class methods. I worked on getting rid of those methods and replacing them with more generic methods that can be used for all sensor sampling types.

I have also been researching the best method for creating a thread in android classes. The SQLite work I’ve been doing is almost finished except it’s writing to the database too quickly which is making my primary keys fail. I am looking for best practices for writing to a database in a thread. In Seshat (our old application), we had a feature where the user could select how often they wanted to write to the database (in seconds). I plan to move this feature to Field Day but I want to research the best approach. In Seshat, we used a runOnUiThread method. This may be the best approach but there are other options to consider — Service, AsyncTask, Handler, Runnable, etc.

Back to Android and SQLite

with No Comments

Since the beginning of the semester, I’ve been working on Field Day again and somewhat sorting out git. At the end of last semester, we were having trouble with git and we’re still trying to figure that out. We’ve decided that we’re all going to stick on the same Android Studio version and upgrade at times we decide as a group. Nic has the most updated version 1.5, so we’re all going to stick to that.

The latest code push was code that broke my Field Day setup, so all of the code I’ve been working on hasn’t been pushed yet. I’ve started a new branch of our git repo called SQLite work that I’ve been committing my changes to. As a result, I’ve been learning a lot of git and branching and I think this may be the way to do so we don’t end up with broken code in our ‘master’ branch.

The part of Field Day I’ve been working on is the SQLite database. Previously, we wrote all of our readings to a CSV file but that gets quite messy. So, in this new application we’ve decided to go to SQLite. I’ve written a class called ReadingsDatabase which is a SQLiteOpenHelper class. This is used to create, delete, upgrade, update, query, etc the database. I’ve also written an interface that will be used to communicate between the Sensor Fragment classes and the main activity. I’ve been using the ‘Built-In Sensors’ fragment to test because we haven’t created any of the other sensors to test yet. I believe it is working, but the problem I’m running into is the class is writing to the database too quickly which is causing the primary keys I’ve set up to fail. I’m looking into a way to slow down the writing to the database. Once we figure out the git problem, I’m going to push my code.

Lots of Android Fragments!

with No Comments

This past week I have done a lot with the Android application, Field Day. I’ve implemented the basic architecture that I think we are going to use.

It makes use of Android ‘Fragments’ class. Fragments were introduced to make it nicer when building UIs. Before Fragments (introduced in API 11), whenever a new screen on the Android device was shown, a new Activity was created. Even if only one thing on the page changed, you needed to create a new activity. That meant that there were many lifecycles that needed to be tracked and maintained. That’s drained the battery and made the code more complicated than it needed to be. The Activity lifecycle is complicated. With Fragments, you can easily swap out parts of a UI and still stay in the same lifecycle. The activity that created the fragment controls its lifecycle. There’s no extra ones that need to be maintained.

Right now, there are only about 3 activities, and 5 or 6 fragments. The main screen buttons, and when you click on ‘Take a Sample’ or ‘Lab Notebook’ are all fragments with one activity maintaining them. Once a specific sample type is clicked then the SensorSample.java activity is created. There are parts of the SensorSample activity that are going to be the same no matter what type of sample we are taking. That’s why those are also fragments. Code/UI elements that will be the same for all — Geocoordinate, for example — are on the SensorSample activity’s layout file, but there’s a FrameLayout in there that can be swapped out depending on the type of sample. It really helps with stopping the creation of repetitive code.

In the SensorSample activity, I’ve already implemented a LocationListener and Manager that will listen and update the user’s location. That will be used for the Geocoordinate. It doesn’t get written to the database yet, but I have implemented in both the code and UI so it shows up and does change. I tested it!

1 2 3