In today's lecture, we're going to be looking at time series and date functionality in pandas. Manipulating dates and times is quite flexible in pandas and thus allows us to conduct more analysis such as time series analysis, which we're going to talk about soon. Actually, pandas was originally created by Wes McKinney to handle date and time data when he worked as a consultant for hedge funds. So it's quite robust in this matter. Let's bring in pandas and numpy as usual. All right, pandas has four main time related classes. Timestamp, DatetimeIndex, Period, and PeriodIndex. First, let's look at timestamp. It represents a single timestamp and associates values with points in time. For instance, let's create a timestamp with a string 9/1/2019 at 10:05 AM. And here we have our timestamp. Timestamp is interchangeable with Python's datetime in most cases. So we just do pd.Timestamp and pass in the string. We can also create a timestamp by passing multiple parameters such as year, month, hour, date, minutes, separately. So pd.Timestamp and we just pass these in order. Timestamp also has some useful attributes, such as isoweekday(), which shows the weekday of the timestamp. Note that 1 represents Monday and 7 represents Sunday. So we create our new timestamp and then we just say isoweekday(). And this is actually quite commonly done to just get a simple numeric value. You can find extract the specific year, month, day, hour, etc from a timestamp as well. So we can create some new timestamp and as we're using it just do .second and see how many seconds there were, who are the second parameter space rather for that. So suppose we weren't interested in a specific point in time and instead we wanted a span of time. This is where the Period class comes into play. Period represents a single time span, such as a specific day or month. So here let's create a period that's just for the month January 2016. So we do pd.Period('1/2016'). You'll notice that when we print out the granularity of the period is M for month, since this was the finest grained piece of data that we provided. Here's an example of a period that's March 5th, 2016. So we just do pd.Period, pass in our date, and we'll see that it's got the granularity of day. Period objects represent the full time span that you specify. So arithmetic on period is very easy and intuitive. For instance, if we just want to find out 5 months after January 2016, we simply plus 5 to it. So pd.Period('1/2016') add 5, and here we go. From the result, you'll see that we get June 2016. If we want to find out two days before March 5th, 2016, we simply subtract 2. So create our new period and subtract 2. The key here is that the period object encapsulates the granularity for arithmetic. The index of a timestamp is a DatetimeInd. And let's look at a quick example because this is really quite handy when you start to deal with time series data. First, let's create our example series t1. We'll use the timestamp of September 1st, 2nd, and 3rd of 2016. When we look at the series, each Timestamp is the index that has a value associated with it, in this case, a, b, and c. So here we do pd.Series, we're going to give a list, abc, so these are our data values. And then our indexes and array next to it. And we're going to do create three Timestamps. One is going to be the first day of September then the second day of September then the third day. Let's take a look at what this creates as a series. So looking at the type of our series index, we're going to see that it's a DatetimeIndex. So we do type(t1.index). Remember the d-type above is referring to the data values in the series not the index. So here we can see that the index is a DatetimeIndex. So similarly, we can create a period-based index as well, which I think is really quite nice. So we'll create some new series, def, and we'll just create our period objects in that list for the index. And so looking at the type of ts2.index, we'll see that it's a PeriodIndex. Now, let's look into how to convert to Datetime. Suppose we have a list of dates of strings and we want to create a new dataframe. So I'm going to try a bunch of different date format. So here I'll say d1 = ['2 June 2013', which is common August 29, 2014', that's also common. '2015-06-26'. And then '7/12/16']. So if you take a human entered data, this is what you're going to get. Or if you take data that comes from multiple different systems. Let's say you're joining dataframes and you're bringing in date fields from multiple places. And now let's just create some random data here. So we'll just create a new dataframe, just with some random integers. And then we're going to set our index to d1 and some columns. And let's take a look at that. So using pandas to_datetime, pandas will try and convert these Datetime and put them into a standard format. because this is ugly and horrible to deal with. So pandas has these nice built-in to_datetime to try and figure out the format for us. So let's take a look with ts3. So to_datetime also() has options to change the date parse order. For example, we can pass in the argument dayfirst = True to parse the European date if we think that our data is going to be in that format. So here I'll take '4.7.12'. You see there's some ambiguity here, what's month and what's day. And well, even what's year, I guess, it could be an ambiguity here. So we could just say dayfirst=True. So Timedeltas are differences in times. And this is not the same as a period, but it may be it feels like it at first, but it's conceptually very similar. So for instance if we want to take the difference between September 3rd and September 1st, we get a Timedelta of two days. So we can actually just take a Timestamp- another Timestamp. And the resulting object is a Timedelta and here you see it says that it's 2 days. It's printed out a nice human parseable thing. But the object itself is capturing that it's these 2 days. So we can also do something like find what the date and time is for 12 days and three hours past September 2nd at 8:10 AM. So we don't have to have these round number. So here we've taken a Timestamp which is a very precise. It's 8:10AM. And here we just want to add two at a Timedelta of 12 days and 3 hours. And we can see what that time is. So offset is similar to timedelta, but it follows specific calendar duration rules. Offset allows flexibility in terms of types of time intervals. Besides hour, day, week, month, etc., it also has things like business day, and end of month, semi month begin, etc. So very non-traditional time series, but things that we would use in business all the time. So let's create a timestamp, and see what day it is. So we'll create some new Timestamp and we'll see which weekday it is and this is now the sixth, so this is from the ISO. So now we can add the timestamp with a week ahead. So we can take that Timestamp and then we can say we want to add an offset of one week. So now let's try and do this to month end with month end. And this would give us the last day of September. So here we can take a Timestamp and then we want to add some offset to get to the month end. And so now we actually have the month end date. Okay, next let's look at a few tricks for working with dates in a DataFrame. Suppose we want to look at nine measurements, taken bi-weekly, every Sunday, starting in October 2016. Using date_rang, we can create a DatetimeIndex. In date_range, we have to either specify the start or the end date. If it's not explicitly specified, by default, the data is considered the start date. So then we have to take the specified number of periods, and a frequency. So here I'm going to set it to '2W- SUN' which means bi-weekly on Sunday. So like regex, there's sort of a mini language to describe these periods. And the pandas documentation, if you get really dealing with time series data, we're not going to in this course. But if you end up dealing a lot with time series data, the pandas documentation is phenomenal. So here we'll do dates = pd.date_range. We set our start, we say we want 9 periods. And this is our frequency, and then look at the DatetimeIndex that comes out of that. And we see that it's well set up for it. So there's many other frequencies that you can specify. For example, one common one is the business day. And so here we just set the date_range. We give it the date, we set the periods, and then we say freq= 'B'. Another common one is quarterly but setting the start of the quarter in a specific month like June. So here we just set the periods to 12 and then our freq= 'QS- JUN'. Now, let's go back to our weekly on Sunday example and create a DataFrame using these dates, and some random data, and see what we can do with it. So dates = pd.date_range. We're going to set the periods=9, and the frequency equal to two weeks starting on Sunday. And now we're just going to fill in to that some data. So I'm going to create a column called Count 1. We'll make these some random numbers and a cumulative sum of these. And then Count 2 as well, and set the index to dates. Okay, so there's our dummy data. So first, we can check what day of the week a specific data is. For example, here we can see that all of the dates in our index on a Sunday. Which matches the frequency that we set. So if we say df.index.weekday_name, we see, they all come back on a Sunday. If we had more variable dates as our index, we would see the variation here. We can also use the diff() to find the difference between each date's value. And this is just on the data frame itself, df.diff(). Suppose we want to know what the mean count is for each month in our DataFrame. We can do this using resample. Converting from a higher frequency from a lower frequency is called down sampling. And we'll talk about this in a moment in a little bit more detail in another lecture. So df.resample to monthly, and let's look at the the mean values. And so here we've collapsed the dates on a monthly cadence. So we've got this kind of date perspective on how to do that. We haven't had to write our own group by and figure it all out ourselves. We're able to use the resampling built-in. Now, let's talk about datetime indexing and slicing, which is a wonderful feature of the pandas DataFrame. For instance, we can use partial string indexing and the key here is that it's actually based on string indexing to find values from a particular year. So if we're just interested in 2017, we can say df['2017'] and that returns to us all the values that were from that year. Or we could do it from a particular month. So df['2016-12']. So remember, these are all string values that were matching on. We can even slice on a range of dates. For example, here we might want the values from December 2016 onwards. So we can just say we want df['2016-12':] onwards. Okay, so that's how the date functionality in pandas works. It's kind of a whirlwind tour we've done here. We're going to talk about resampling in another lecture. And that's maybe going to be a little bit more clear how we use this. Again, if you have to deal with a lot of datetime data, this lecture will become important to come back to to refresh and understand. And also the functionality of pandas with respect to datetime is really quite phenomenal and the documentation describes it in more detail.
