Hello everybody, welcome to Python for Everybody. This is another work to code example. You can download the sample code zip file if you want to follow along. The code that we're working on today is what I call the geodata code, and that is code that is going to pull some locations from this file. We're simulating or using the Google places API to look places up and so we can visualize them on a map, and so this is the basic picture. If we take a look at this where.data file, it's just a flat file that has a list of organizations, and as actually was pulled from one of my Mooc surveys. We just let people type in where they went to school, and this is just a sample of them. So this data is read in by this program geoload.py, and if you recall, this Google geodata has great limits. It also has API keys which we'll talk about in a bit too. So the idea is, this is a restartable spider like process, and so we want to be able to run this and have it blow up and run it and start it and not lose what we've got. So this is unlike, some though, so we're not now using a database, as well as an API. But in order to work around the rate limits to this API, we're going to use the database with a restartable process. Then, we'll make some sense of this, and then we'll visualize this, but in the short-term, let's start with the geoload.py code. Geoload.py, take a look here. So, a lot of this hopefully by now is somewhat familiar to you, Urllib, json, sqlite. I mentioned, that the Google APIs, these used to be free and did not require an API key but increasingly, they're making you do API keys for especially new ones. So what happens, you can go to your Google places. I mean, go to Google APIs, and get an API key and you can put it in here. It'll be this long, big long thing, it looks like that. Then if you have an API key, you can use the places API. I've got a copy of a subset, not all of it, a subset of it here at this URL. As a matter of fact, you can just go to this URL in a browser, and it will tell you a list of the data that it knows about. I made it so that, that does the same basic protocol with the address equals as the Google places API. So, this will just change how we retrieve the data. Either retrieve it from my server, nice thing about my server, it's got no rate limit, it's really fast and you're not fighting with Google all the time. It means that perhaps, if you're in a country that Google is not well supported, you can use my API. I mean that's really strange, but somehow my API is more reliable and available than the Google one. But, it's true. So we're going to make a database. We're going to do a create table if not exists, and we'll have some address, and we're really just caching the geographical data. We're gonna cache the json. One of the things we do when we build these processes is we tend to simplify these things and not do all the calculation and parsing the json, just load it and get it in, and load it, and get it in. Fill the data up in this database, so that's what we're going to do. Because Python doesn't ship with any legitimate certificates, we have to sort of ignore certificate errors. We're going to open the file, we're going to loop through it, pull out the address from the file, and we're going to select from the geodata where that addresses the address. Let's move this in a bit. We're going to do a select, and pull out that address. The idea is as if it's already in the database we don't want to do it, so we do a fetch one and pull up that first thing that will be the json right there. If we get that, we'll continue up otherwise we'll keep going. Pass, just means don't blow up. So we accept and we just do a pass that's a no. We're going to make a dictionary, because that's what we do for the key-value pairs. Everything you've seen so far, I've used constants here, but because we may or may not have an API key, query equals and then that's the address and then the key equals and then the API key. If you recall URL encode adds the plusses, and question marks, and all that nice stuff. We're going to retrieve it, we're going to read it and decode it, print out how much data we've got and add account, and then we're going to try to parse that Json data and print it if something goes wrong. As we've seen at this top level of this json data from this Geocoding API is an object, which we'll see a little bit of it in a bit. It has a status field in it, and the status is okay, if things went well. If the status is not there, that means our JavaScript is not well formed, or not how we expect it. If the status is not okay, or not equals zero results, then print out failure to retrieve and then then quit. Then we're simply going to insert this new data that we just put in. Then we're gonna commit it, and every tenth one, this is count mod ten, we're going to pause for five seconds. We can hit control C here, and then we're going to do the geodump. So let's just run this, geodata, Python. So let's do an LS. So we do have, let's get rid of from a previous test geodata LS Sqllite. So we'll start with a fresh set of data, and run Python geoload.py. Of course, I'm always forever making the mistake of forgetting Python three, so you can see that it's running. It's adding the query and in this case I don't have the API key. It's putting the pluses in and that's this part here with all the pluses, that's the URL and code. You notice that it's pausing a bit, depends on how fast your net connection, this may or may not go so fast, but this is not that much data, so it should. It's like only 2,000 3,000 characters. So it's working and talking to my server. The interesting thing here is I can blow this up, I'm going to hit Control C. In Linux you'd hit Control C and on Windows I think you'd hit Control Z. Depending on which shell you're working in. But I'm going to hit Control C, and you see I sought blew it up. That causes a keyboard interrupt traceback. I do an LS minus L, you can see that now there's Geodata is there. Now, in the name of restarting, I will restart this. You will see that it checks, and skips and so it runs this code here, where it's right here. It grabs it, and finds it in the database. So you'll see it's found in the database really quick, chop, chop, chop and go really fast. Then it'll go back to catching up where it left off. So all those up there, they did not actually re-retrieve it, because it knew about those things. Now it's catching up, and doing some more, and doing some more, and doing some more. Then I'll hit Control C, it has a little counter in here that basically, if it hits 200 it stops, and you have to restart it. You could actually change this code, you can make it so it didn't sleep. Doesn't hurt to sleep for like a second after every 100 or so if you want. You can change that code. Now let's just hit Control C and blow it up. LS minus L. There is another bit of code, and this code it's always good to write these really simple things. Now we're going to now import Sqllite and json. We're going to connect ourselves up. We're going to open, except this is a UTF eight. Because it's a UTF, we're going to open this with UTF eight and we're going to read through. In this case, we're going to decode. You just select start from locations, and if you recall, locations has a location and a Geodata. So the sub zero will be the location and the sub one will be the the geodata and we're going to parse it. Convert it to a string and then parse it. If something goes wrong with json we'll just keep skipping it, or check to see if we have the status in our json. Let me run the Sqllite browser here. File, open database. Let's take a look at what's in this database. Where we code three geodata to data SqlLite. So this is our data, we've got so if you, I'll make this a little bigger if I can't, can I make that bigger? Yes, I can show us much. So you can see that these are the addresses and geodata that's just json. So that's the json that we've got and it retrieves it. So this is a really simple database, there's just sort of spidering process run run run. But now we're going to run the geodump code, which is going to read this and dump this stuff out, and print where.js, so it's going to actually parse this stuff. That's code we've seen before. So we're actually reading it and this line goes into the results. Results is an array so if we go into results, results in an array. We're going to go grab the zero item in that array, then we're going to go find a geometry, and then location, and then lat and long for the latitude and longitude. Then we're also going to take the actual address out of the formatted address right here. So, in this bit of code, we're actually parsing the json and we're going to clean things up, get rid of some single quotes. This kind of data cleaning is just stuff after you play with it for awhile, you realize oh my data's ugly, or does this. Now print it out, then I'm going to write this out. I'm going to write it into a JavaScript file. So the Javascript file is this where.js. I'll show you what it looks like, it was going to be overwritten. This is the one that came out of the zip file. It'll have the latitude, the longitude, and we're going to use JavaScript to read this. In this where.html file, it's going to actually read this right there, and pull that data in. That's how we're going to visualize. I'm not going to go into great detail on how the visualization happens, but that's what's happening. So we're going to write that, we're going to actually write this to a file. Let's go ahead and run this code, and say Python three, geodump. Okay. So it wrote a 120 records to where.js. So if we look at where.js, this is now the new data that I just downloaded moments ago. It says open where.html in a browser. Now this you'll need the Google Maps API, and you might not be able to see this, depending on where you're at. But, here you go with the Google Maps locations. I think if you hover over this, you can see. You see the UTF why we're there in that particular thing, why we had to use the UTF eight when we wrote the file. So that we didn't end up with trouble writing the file out. There you go and so that is a simple visualization. Just a simple visualization and wrote this where.js. If you are smart with HTML and JavaScript, you can look at this where.HTML file, it's really just reading through a bunch of data and putting the points. That's all there is, but I'm not going to go through that, at least not in this. I hope this was useful to you and thanks for watching.
