distance

Shawn is always worried about turning our little unit off when he gets near his house.

Since we need the ability to set a perimiter around a point - I thought it was a good time to learn Functions within SQL Server, and to implement the Haversine formula for distance between two points.

Read up on Functions - and got to work.

we now have one for the derivation of distance between two points.

from there we have one for all points within a specified proximity

and from that one - deletion of points within another point and some distance.

The latter is mostly for house keeping.


Now - problem is it takes quite a little bit of time [several seconds] to process a few thousand rows - which is a little slow.

So I might just have to re-write it with CLR.

----

I also drove to Gerringong after uni and from there to Sydney. Will be quite interesting to see the playback of the trip.

And as Shawn's post already states - this was the first drive that was 'visible' live. Now we just have to work out the quirks.

I will have a look now at the performance of getNextPoint vs getLastPoint, see if I can somehow improve it.

map update

Ok, I have got the following working in the map.php

Single Unit Live Tracking,
Single Unit History

• Points (Added one at a time),
• Line (added all at once).
  

I need to do this for multiple devices,
I need to look at the GIcon, and having different ones for different speeds,

I have two issues with the above, I am currently hardwiring the initial start position of the map, if I tried to do this using the first data point, I would get NO map...still trying to figure it out. Also, I need to do a setBounds on the Map, so you can see all the data.

Roman has pointed out there might be a lag between our map and actual. I am sure this comes down to the use of getNextPoint and the amount of time it takes to process. I could change this to use getLastPoint, only issue is we may get holes in the data...will try it out but.

map.php generating maps

Well

Over the last week I have been hacking away at our map.php page. My initial plan was to use a combination of straight php odbc call to our DB and then display this information using Google Maps API (javaScript). Now, this works fine until you need to loop through the database to get live updates of the current position of a iTrac Device...since php is server side, that is, it all gets processed on the server before reaching the client, my design fell over. Instead of getting then next point in the DB, I was getting the same point over and over again. This was even worst when trying to plot a history of points.....BIG PROBLEM

Solution, ajax xml parsing and php output to xml. So, I now call a php script I have written, then read in the output using ajax GXmlHttp. This way, I can query the DB, using php and then read in the results.

So, Over the last couple of night I can now do:

• A live display of a iTrac unit and see were it goes,
• A history display of a Unit to see where it has been.

In both cases, this is only as Points, I need to get the PloyLine display working next.


Thats it for now

Shawn

Quest for perfection

Tonight I continued my quest for getting the EACH and EVERY point displayed on the map.

I didn't see Shawn today, as he stayed home to tinker more with PHPme[PHP mapEngine]. So he kept the phone - at least until wednesday.

I wanted to test the new software I wrote on the weekend, so I used my phone to give it a test run.

Good news and Bad.

The good - every little point is accounted for.

Also a feature addition requested by Shawn, is to have the unit remember the device with which it was last paired, thus preventing the user from needing to find his/her device every time.

So once the device is started for the first time - it will start looking for new devices.

Once you select one - it will add it to the RMS [Record Management Store] on J2ME - (worst data storage method/api ever invented)

Next time a unit starts - we check the datastore, and just connect to the same device.

Voila!

---

My NMEA parser works like magic - and doesnt skip a beat. Which is great news - given the bugginess of my previous attempt. I used to clean the strings generated by the GPS, as I couldn't synchronize properly - now, it works like a charm.

I did more research on it - and one of the papers i referred to is:

http://www.visualgps.net/Papers/NMEAParser/NMEA%20Parser%20Design.htm

which has a very helpful Flow/State diagram.

---

now for the bad.

Even though the parser didn't skip a beat, and one of the things i did to simulate a tunnel [or any loss of gpr + gprs signals] is every 10-15 seconds i put the gps unit inside my filing cabinet.

This lead to two things.

1. the bluetooth signal was lost by the phone [not the connection - as the timeout is too great, just the serial dump has ceised to exist. as soon as I would take the phone out - the connection [bluetooth] is resumed and it now gets a new UTC time from the sattelites and sends the new string.
Mind you - the utc time is now several seconds past, which in turn results in a 'gap' if i was travelling.
This being a given scenario for say a tunnel - ALTHOUGH - in the tunnel, we will still be able to maintain GPRS connections (anywhere in sydney anyway). So the UTC time will continue to be updated on the device, and thus sent to the server.

2. the 'old' Jargo prototype/mashup that i had before - wouldn't synchronize with the device past the initialization, or first contact. This has a nice 'buffer' result.
Where-by as soon as turned a gps off - it[jargo+phone] continued to send co-ordinates to the server as if it were still moving. So this gives me an idea on how to solve the following problem/handicap that i discovered which is ALSO a result of points going missing.

---

whenever the phone/device sends a request to the server, that request has to be processed. then returning a 'point added' to the phone.

this is a nice little problem, as

the device can not continue to send other points until it verifies that the point has been added, as the server will hold the device 'on a leash' until it inserts the data into the database

this has two solutions - one simpler than the other.

1. Create an array/list on the phone, into which new points get added as they come off.

This array can then be queried and have the item's "popped" and sent to the server.

The server can then accomodate for an array of points to be received as 'one' statement.

This is more of a 'controlled' buffer - than the 'bug' one I've had previously.

One mis-direction [i'm sick of the word 'problem'] that i can forsee - is a direct effect this will have on 'real-time' display of a point. Until all points are sent by the device, it is no longer where the data it just sent says he is.

this brings me to solution number:

2. Every time a device sends a request to the server, automatically respond, and THEN deal with the query.

This allows the database to queue all the insert statements, and add them at it's earliest convenience.

This will also lead to a much faster interaction with the server, and not leave the Web interface tied up.

---

All this brought me then to another quandary [i decided to use a thesaurus for that one].

The performance hit on the web server, and what is the actual load that it can take 'comfortably' - given either scenario.

I coded up a little Load Test. results of which can be seen below.

They will show us the results for a 'slight' load (100 users) slowly [every 5 seconds] increasing how many requests get sent to the server.



We can see that the processor is getting raped [figuratively speaking]

yet the requests that are queued are fairly stable and mostly sit under 20.

mostly if we look at the avg. response time we see that it also is quite consistent

on the other hand if add a few more users [say 10 times - up to 1000]

we get:


Which shows that our avg. response time is much higher - at nearly double the rate.
We also see that the processor is also enjoying itself.

Now.

I have to code the 'new approach' to receiving user requests - and then i'll run some more load tests, to see if there's any difference.

Jargo.v2

Great news came in on Friday - Shawn plotted points.

We picked some random PID's to have a look at. It was great seeing points come up for the first time



Although we noticed a small problem - it would skip certain points and not be a uniform representation of the data.

So I investigated that yesterday. I first of all looked at the raw data and the speed at which he travelled to make sure he didnt suddenly stop.



The graph is his speed over a period of 40 minutes. [time starts on the right].

we can see that in the period that i show above - his fluctuation in speed cannot be responsible for missing data.

so i had a further look at the data, and saw quite an immense discrepency between the time points were added to the server, and the UTC time recorded on the devices. i expected a direct corrilation between the two. Yet what I got was a fluctuation of between -3 and up to 20 seconds.

then there were no points for several seconds and some times up to 10 seconds, which would account for missing data points.

So I put the missing points to two possible variables.

1. Woop Woop - places that Shawn drives around - usually have not the best GPRS reception. So that is a very feasible problem cause and would explain the missing points.

2. Software that I wrote for the phone doesnt sync. properly the baud rate of the GPS device.

So today I re-designed it from the ground up today in order to be more precise and hopefully sync properly.

Astoundingly - it[software - aka Jargo {stands for GPS Aggregator}] now reports accuracy up to 1 thousandth of a second. Only way to check that though - is to load it up onto our test phone [which Shawn has been carrying around] and give it a whirl. No matter how accurate it is though, will still have to account for the delay in communications.

CSS + ASP.NET

i have not much loving for css and asp.net.

have been working on setting out the menu for the map and the ability of css representation to be consistent across browsers is fantastic - its non existent.

couple this will asp.net and you have the perfect way to spend your weekend designing menus and sitelayouts.

Maps generated using php and our DB

Well, over the last 4 days I have been doing about 3 -> 4 hrs a day on getting my map.php code to work. At present it works kinda like this

1. Receive a function and a uuid via $_POST,
   
2. Query the DB to get the current positional information using the uuid,
3. Create a map and centre it on this info,
4. Add a point which has this positional information stored with it, so when user clicks on it, you get Lat,Long, Speed and Direction.
5. Update the map...at present it just adds 0.001 to the lat and long. I need to change this so it re-runs the DB query and gets new position.
   

Once I get 5 sorted I am going to move onto displaying polylines

Shawn

more features!

had a meeting with daniel today.

went over the deficiencies of our current non-existent user/tech manual at first. Then we got into talking about possible 'markets' for our project, and a way to perhaps maximize the audience.

daniel asked for the following features to be implemented as part of the portal:

- Generic Database connection/interaction

This will be responsible for allowing our portal to utilize other databases, to prevent us from being locked in an MS-centric environment. Databases to implemented:
* MS SQL [what we run everything on now]
* MySQL
* Oracle [Andrew's favourite, so we shouldn't have any problems with this one]

I do not forsee too many major problems in implementing the cross db support, as our schema is very simple in nature and should flow quite well across the different databases.

One design aspect that will be changed though - some of the actual logic will have to be taken outside the data tier and put either into the presentation [portal/webserver] or taken out into a separate logic entity/processing entity.

- LDAP/Radius Authentication

Provide the ability to map users against enterprise credential systems such as Active Directory. This one should be interesting - as at the moment, as far as I know, none of the group members have had any experience in developing LDAP encompassing software. Andrew is designing the Membership Provider at the moment, so I'll have a talk to him regarding seeing what we can do with that, or perhaps we'll need to write something more scaleable/generic.

- Unix Hosting of ASP - aka mod-mono

Daniel wants to see if we'd be able to put this up on a Unix server for hosting, thus eliminating the reliance on MS products once more. I shall have to do more research this weekend to find out the feasibility of such a proposal, given the toolkits that we 'might' be utilizing to accomodate the LDAP-driven authentication.

Group Work + Database

This weekend has been great for me.

Saturday I spent researching SQL Server and its take on stored procedures, transaction control and CLR.

then did some research regarding database design, and proceeded to implement what I was hoping Andrew to have complete at the beginning of this session.

I have written several stored procedures which allow Shawn to use the database for the PHP Mapping Engine he is building and pull out points for specified UUID's/Devices. Then a couple of procedures to pick up the next point in a record set for displaying the points.

Then I proceeded to split up the all the GPRMC strings that we've generated since the use of gps + mobile combo. Some of the data was Shawn's trip on the mountains - so that will be interesting to have a look at once plotted. The splitting of the string was done by a parser class I hacked together that spits out decimal lat+long's as well as km/h and other attributes of an RMC string.

Now we have a database with which we can work for the display of points.

It is now necessary to write more procedures to make Shawn's job easier in locating the necessary data for a device and any criteria such as subdivig a drive into separate trips.

I hope that Andrew will have a Membership provider finished by this Wednesday, as I've taken a lot of what he was supposed to do off his back.

Once that is complete, I can commence integrating the PortalSite with the database.

Main features I want to focus on initially include user creation, logins and profile settings.

In conclusion - good news. I spoke to Karen Williams today - and our WebTrac device has finally been ordered.

So unless it gets stolen/broken/misplaced along the way - we should have it in use by next week.

Site layout + WebTrac + Database

Over the week I have compiled the look of the website. I am designing the layout with 'div' tags in order to then apply CSS pages atop.

There will be a slightly different view for a logged in user, than that of an anonymous one.

Am still aggregating data with the mixture of my phone + gps unit, as the WebTrac unit has yet to be ordered, yet alone delivered; which kind of defies the idea of an 'all-in-one' solution.

Anyway - the data is still being appended to a single table of values, since the database is another week late. Andrew proposed a new date for completion of Wed: 16th August.

Hopefully the following are in place in order to conduct further development of the portal:

- membership provider for users
- data from the single table, split up over others to accomodate the schema then in place
- functions for:
+ checking validity of uuid/imei, if not valid - create table
+ appending data (uuid/imei, rmcString, addedDate)
+ reading points ([arguments defined by Shawn - as he'll be using this one mostly])

I have supplied Andrew with several books on Sql Server, ASP.NET membership providers as well as C# - hopefully they'll be of value.

Collecting Some Data

Well

Last weekend I took Roman's phone, our SIM card and one of the GPS units for a little trip. Started off at the bottom of Mt Ousley, then drove to the snow :-) Had a couple of stops, and we had some losted connections in a couple of area's so there might be some holes in the data.

I then skiied on the Saturday, but left the gps unit off and locked away....didn't want to break it, been 4 years since I boarded :-(

On the Sunday, however, I used the unit pretty much all day. Am looking forward to creating some "replays" of some of the runs I did, went all over the mountains...

Might edit this later and include some links to the results.

Shawn

Generic Portal Page Design

Well, last post I said I wrote down some idea's on the portal design and some functionally...

What I have next is what I see each page needs to contain:


{Script Header/Key/vrl/DTD}
{Top Nav Definition}
{Login Script/Code and check}
{Interaction Code/ Requests/ Left Nav "features"}
{Display - Map}
{Interactive/interaction Code}
{End/Footer}


Now, here are some more notes on a couple of them...

{Display Map}

• createMap
• addControls
• setCentre(Based on unit(s) location)
• displayCurrentLocation as Marker
• UUID -> Use a Nickname,
• lat/long (georeferencing -> Address - assumes land based),
• Current Speed,
• Current Direction,
• Reading Type,
• Time reading Taken, and
  
• Time reading received.
• Different Markers for different states
• Speed Ranges,
• Reading Type, and
  
• Age of Reading.

{Interactive/interaction Code}

• onclickMarker
• Display some of the above info in Tab's
• History Display
• Date range/Number of points
• Display as markers, OR
• Display as line
• Geofencing
• Click, define radius,
• Inside or Outside alert,
  
• Set/Save to DB,
• Define alert method,
• Marker/icon colour,
• popup,
  
• email, and
  
• SMS.
• Speed Alerts
• Define speed (Max/Min)
• Set/Save to DB
• Define alert method
• Marker/icon colour,
• popup,
• email, and
  
• SMS.

Roman and I need to define the API which we are going to use to interact within the portal, while we need to let Andrew know what functions he needs to create, and what trigger are needed.

Shawn

Dynamic Updating of Maps

Ok

For the last 3 nights I have been trying to get the Map to update the "current" location of the maker. This was me trying to prove a concept that I could run a query and update the Map if the data/location changed.....after 3 days = 6 hr's, I have finally done it...man... I hate it when I make silly mistakes.

I had the following code

define some vars
createMarker function
onClick function
updateMap function //the bit I was trying to use as Proof Of Concept

create Map function


Now, I had declared the "map" within a function, I was then trying to update this map from another function....dooh...

IT DIDN'T WORK....

WHY....

SCOPE...

once I decalred the map in the "main" it worked a charm. Now I can move onto actually a bit more design...I wrote some ideas down before my CSCI318 lecture today...will post them next. Once I have done that...move onto query of DB, displaying Unit's location...then onto some cool feature we have in mind.

Shawn