Todo Today:
1. Find a way of implementing complex api calls -> mostly done
2. Find a way of monitoring results for error codes and xml parse errors ->flckr sends and error code rather than the page not exisiting, helpful to people using the api for real but not to me!
3. Test!!!
Its three in the afternoon and I’ve been here about 4 hours. Its been a good day full of problems, ideas, and developments. Its also given me the chance to take a look at some of the other projects being worked on.
Things I’ve managed today:
1. Implementing User Input
I’m still in a real development stage of the project but have hocked up some really simple textfields and an output area so that I can see what data is going in and coming out. It also allows me to code in the availability of user input now rather than later.
2. Generating simple usernames
Utilising the above user input it has been possible to generate some simple alternative usernames for when the using the default one fails. Its not fool proof but may fill in some blanks.
3. Integrating the FEEDS.XML file.
The feeds.xml file is basically a list of the api’s and rss feeds I’m using to pull data such as youtube and last.fm. This was a bit problematic as the way flash loads external files is in a way threaded. This is great when loading large files as it will not sow slow down the application but when trying to use a for loop to loop through a section of urls causes massive problems as the next loads before the first has finished loading correctly. To fix this I had to be bit cheeky and probably a real bad programmer but my solution was to use a callback on the loader (eventlistener) when this fires it moves to the next feed/api and loads that info. Not elegant but It’ll do for now.
Things to do:
1.Testing
I want to do some more testing to make sure that all feeds are parsed correctly and I can easily add new feeds/apis to the list.
2. Complex API Callls
Some sites such as flickr and digg require some special parsing as not one but two or more calls are needed to get at the actual data needed. This will be a bit of a challenge but will hopefully be achieved soon.
3. PHP Web Proxy
Flash has a security sandbox much like any other applet/plugin used in a browser to get around this I need to make a simple php proxy service to pull the xml through a localhost which will be allowd.
I’m still confident that I can have this up and working by the end of the week and It would mean great progress if I could. With the data side finished I could then look at the visualisation side of the project, and finally polishing of the application.
Peace!!!!!!!!!!!!!
(Only took half an hour to write)
Today has been productive, a nice test in the morning with an easy 73% pass and an afternoon of m final year project and playing around with terminal, which is always fun.
Things done today for the project
This was really needed for two reasons, firstly to catch any sites that I can’t get a result from and supress the error and secondly and more importantly to try and find some relevant data on that site/service. This second haas more to do with data mining, i.e not finding anything for the given username but trying some variants to pull whatever data is there. This does run the risk of pulling data from a different user but I will make it confiigurable to some extent.
Implementing try…catch in as3 was easier than I expected and after some failed attempts found it wasnt actually possible with the way I was developing the application to call apis and rss feeds, instead I have used an eventlistener to listen for fail url loads.
Personally I actually like JSON now, it makes sense when dealing with php and js or any ajax aware website and now it works in flash thanks to the corelib provided by googlecode and some people (find the poeple and put there names here, credit where credits due). JSON has now been added and parsed into my uniform results object.
Some websites, only a few, dont provide an api or rss feed in any form and I want to try and pull some of this information if possible so I started looking as dom manipulation in as3 however It doesnt look possible to do this. I may have a crack in php and create a little service I can call to provide this functionality.
4. Better Data Mining
As I said earlier the useful part of the try…catch/eventlistener addition is the ability to do a little data mining and hopefully find some information/data from each service. In order to do this I will need some basic user information. I’m currently thinking something like:
These bits and bobs of information should allow me to create some basic username like , firstnameLastnameDOB for instance and email is helpful as some services require it, such as flickr. This is a more work in progress as I need to set up user input first of all and then create a subset of usernames and store in an array to push through when nothing is returned.
So its going well. Im feeling positive for once and it seems like the end of the week I should have the basic data collection and parsing section of the application built. With this done I can focus on visualising the data into a dna structure to show the users digital person/characteristics.
Thats all for now!
Well lets start at the beginning of this mini adventure, since realising that it wouldn’t be possible to use Processing.js to visualise the data I had collected it has also become apparent that although my data was nicely formatted in objects and arrays they bore no resemblance to one another and this could be a major issue as I’m trying to write this so that in the future new feeds can be added from a gui perspective rather than having to rewrite the code each time I add a new feed.
Sensible as can be right?
Yep, but it does mean that in order for a visualise to work I need to have the data in similar strucutres on one another so that I can produce a single engine/class that will work for all without any modification. This has led to a number of issues.
1. Not all rss feeds are the same, some are atom, some are rss 1.0, some 2.0 and some are just xml with not a care for specifications which is a nightmare. So my plan is to find the highest most repeated element and hope that this is the item element such as in atom feeds.
2. JSON parsing in flash is almost non-existent and isn’t really documented once you find the library to use. Not dire but just bloody annoying. Thanks for writing the library tho, I really do appreciate it.
3. Sunshine is nice. Working is rubbish.
Well that last one is just an observation out of my window. Seriously its awesome weather right now for the UK. You know what we are like, constant rain and wind and snow and all that crap, but its warm and sunny and its making me want to go and have a pint!
Anyhow good news. I think I’ve finished the xml import into flash and create an object. It’s currently only tested with one feed but I will do some proper testing on it and hope it works. Still need to look at json, I built a version yesterday off of the php code and its a bit buggy and broken so todays build is a whole new kettle of fish and seems to be working great so far.
Things to do then:
On a side note I finally figured how to dispatch my own events in as3, I tried for like a day on our games module piece to find out how to add custom event listeners and got no where so I feel today has been justified.
Peace!
I’ve been lazy, lazy as lazy can be and Its all my fault, well and the weathers! Damn sunshine.
So anyway apart from the laziness I’ve also come to acknowledge that using processingjs is going to be in reality impossible as no broswers support webGL yet which I wanted to take advantage of to create these 3d dna strands of a users online profile.
In fact only the minefield version, i.e. edge release or firefox supports it and I dont think thats building something that only works on an experimental browser is a good idea. So looks like I’m using flash. This shouldnt be too much of a problem as I am a reasonably good AS3 scripter and have already worked out the majority of the logic in php so it shouldnt be too hard to port this across to as3, in fact I hope to have a proof of concept build done this afternoon and released to my dropbox account. It wont be anything visual just the pulling the data into flash and displaying it upon the stage in some way.
So hope to post in a few hours if not bear with me!
Peace!!
So I’m actually getting somewhere and feeling pretty happy with myself. The api class I’ve build in php allows me to collect data from url, and by supplying parametes such as type, apikey, user_agent I am able to collect the data and parse it into assosiate arrays which I find easier to work with rather than a generic object. I have also compiled a list of the currect data sources so things like Last.Fm, youtube, picasa , twitter and the parameters they need to use and stuck them into an xml file which is loaded the same way as everything else and passing out the links into the api class allows me to return all the data I need. Woop! Sticking in my username gave some results and a few warning becuase I havent yet set up error handlers for when the username doesnt exist at a service and some do not provide an error.
So yeah, getting the data is done almost, I still want to add in the ability to mix up the users names, so obviosuly the user will need to put in their name and username and maybe email etc.
I can user all of these where applicable, username will work for most but if on one site the username is different I can try a few other common ones, firstnameSurnameDob for example or initalSurname and some services actually allow search by email which would be awesomely simple.
All things to consider this afternoon and work on once I have a plan.
Then I just need to start playing with this data, for which I am going to use processing.js to begin with, this is a javascript port of Processing which is something I do love to use when I can.
Also I realised I need a secure login area I guess to maintain and update this xml file of all the data sources and add new ones and set up other pages such as what this is and what it does and privacy policy and all that jazz, so initally I was going to use a flat file architecture but its boring and I dont need something as heavy as mysql so I opted for sqlite3, something I played around with when I looked at ruby on rails.
I’d quite like to look again at RoR development it was fun but I couldn’t dig my teeth into it, now I’m a little wiser I might revisit it.
Anyhow sqlite3, nice and simple, and probably better explained here or here rather than here.
Peace!
Well for a few services actually only two, last.fm and flickr I had to sign up to their api and recieved an awesome api key to use that uniquely identifies me which is nice I guess and hopefully once this thing is finished will allow me to showcase it on their sites somewhere, who knows.
At least it was easier that digg, Digg exposed a users diggs through a simple rss feed which is great, nice and simple however everytime I tried to retrieve any data from it I was given an access denied error which was confusing as I know the url worked as I copied it directly from the address bar. A little digging (ha), swearing and lunch put me back on the right track as it appears that they like to user the user agent string to provide statistics of how the data is used so I needed to simply send a user agent string along with the request. This can be achieved by using
PHP$opts = array('http' => array('user_agent' => 'whatever_you_want_to_call_it',));$context = stream_context_create($opts);libxml_set_streams_context($context);// request a file through HTTP$doc = DOMDocument::load("url_you_are_trying_to_achieve");
I picked this up somewhere and should have noted the address to thank whoever did it but oh well. Thanks random webpage that is somewhere in my history.
Well I’m back, and moving forward. I’ve basiaclly finished making a single class to import the xml feeds from various websites and parse it into an array for use with the visualisation. Although along the way I’ve had a few issues, atom is a pain but is now fairly simple to process and youtube thankfully also expose the response in json so a quick json_decode and a nice simple array of values including that important views/duration data I wanted.
I’ve also had a few issues with the encoding of some feeds including non utf-8 characters in a document with the encoding value of utf-8, this was a simple fix of simply when it fails due to this error simply utf8_encode that bad boy and everything is fixed.
I’m almost at the point that I can start fiddling with the processing.js ( the javascript variant) to try and visualise this data, no idea how well that will work but its an experiment and if its crap I can always use flash or real processing or even open frameworks if I’m feeling ambious enough to try and learn c++ in like 3 weeks as I’d need at least another 3 weeks to build what I want and a week to do the write up.
Hooray I’m at last starting to get somewhere, It turns out thanks to my all powerful friend google that it is possible to get this data out in a a way this is to use the namespace value when iterating throught the children of the simpleXML object, it does seem strange that they dont appear when we print the object but to my knowledge they should still be there and I may have to do some hard coding to get the values out.
Thanks IBM -> http://www.ibm.com/developerworks/xml/library/x-youtubeapi/
So as I said Im getting there and this day has really been slow but hopefully all the issues encountered today should allow me to easily push forward tomorrow.
Also found this the other week and it entertained me and kinda of shows what I’m playing about with the fact that google holds all this information about you and often you, and even I forget just how much of us is now on the net for all to find.
Well I’m moving forward slowly, I’ve been prattling about with PHP mainly because I havent touched it in a while and want to re acquaint myself with it.
So its taken me all day to parse the atom feed from youtube into php using simpleXML and then converting it into an arraymaking sure that multiple entities such as entry are put into sub arrays, attributes from the link nodes are put into sub arrays and so on and it works, allday and some horrible logic but it worked, I then noticed that some data wasnt parsing i.e.
<gd:feedlink href=”http://what-a-load-of-rubbish”>
Which is really annoying as the intersting data I wanted was in these sections such as view count, favourite count, duration etc – these could’ve been really useful to control elements of the piece.
I think this is because of PHPs naming convention ie things like colons or hypens cant be used in variable declarations which is what simpleXML is apparently trying to do! So back to stage one, also I found a really simple script to parse the simpleXML object into an array which would have been awesome about 2 hours ago, though it was fun figuiring out the logic and all that junk.
I’m going to start muddling about in Flash I think and see If I can apply the same logic as before to parse the atom feed to a usable array and hopefully parse the cool bits of the data that I actually want. Dont get me wrong I’m not scrapping the php and javascript version in fact I still want to use that as the primary idea, Im just getting a little frustrated with simpleXML and need a change.
Also Processing has a javascript port that may be usable for the visual element of the piece, it can be found here: http://processingjs.org/
Peace!
