In our last team meeting on Friday, I mentioned taking a look into nonlinear regression as a possible way to analyze our data. Matt looked into this a bit and found a very interesting video by Zico Kolter, an Assistant Professor of Computer Science at Carnegie Mellon. I only watched about the first 10 minutes of the hour long video, but to say the least, it was already filled with very useful content. For example, one of the first images he showed was a plot of Peak Hourly Energy Usage vs. Temperature data in Pittsburgh over a four year period. This was strikingly similar to a scatter plot that Matt created that plotted nighttime kW usage vs Temperature.

According to Zico Kolter's course website, one of the goals of the course is to "address issues regarding the prediction, modeling, and control of electricity". It is exciting to see that we are not the only ones involved with this and that there have been efforts made that we can build off of. Although the video was a bit difficult to understand conceptually, he has some earlier videos of the same course that I plan on looking into. The link to both the course website and the YouTube video are below. 

http://www.cs.cmu.edu/~zkolter/course/15-884/index.html 
https://www.youtube.com/watch?v=wof0eipWP1I 

I started off the day by looking into Poisson Regression, which seemed like a reasonable transition from looking at Poisson Distributions. Poisson Regression is a form of regression analysis that is used to model count data and contingency tables. When I looked into the details of how and why Poisson Regression is used, much of the information I found was difficult to understand and abstract. I'll definitely  bring up this topic during tomorrow's meeting with Matt to see if he thinks this can be useful for us, but in the meantime I will learn about regression analysis.

I looked on the Internet for a source that had lessons on regression, and I found an on-line course from Penn State University dedicated to Regression methods. This looked like a good place to start for me, and I immediately began reading the notes on the website. Hopefully learning more about statistics as I look at techniques to fit data will allow me to better understand what I am dealing with. I looked at the syllabus for the course and there are a select few chapters that I hope to look at in the future, specifically the chapters on Model Building, Regression(logistic, Poisson, nonlinear), and Data Transformations. Today I covered a few sections on the basics of linear regression and over the weekend, I hope to have the areas I mentioned above covered so that I will be able to implement these areas in a script.

I started off today by taking a brief look at the temperature data that Matt retrieved. When I asked Matt about his initial findings with the temperature, he told me that there was somewhat of a correlation between power and temperature. However, he is unsure as to whether the data is faulty or if there is a deeper correlation between the two. I plan on looking more into the temperature data later on when I have more statistical tools at my disposal. I then decided to look a bit into the Poisson Distribution and Poisson Regression that Matt mentioned on his blog.

The Poisson Distribution is a model that defines the probability a certain amount of events takes place in an interval given the average amount of times the event took place. There are a few restrictions that determine whether or not a Poisson Distribution can be used. The events that occur must be independent of one another, the rate which events occur must be constant, and the probability of an event occurring in a small interval is also proportionate to the length of the interval. Thinking about how this can be applied to our energy data, it seems to me that one way that Poisson Distributions can be implemented is by separating dates into groups which use similar amounts of energy (seasonal, monthly, bimonthly, etc.), and creating a Poisson Distribution for each of those groups. In this way, we can say that the rate at which events occur is a constant value.

One point of interest with these distributions is to make sure that the groupings neither have too many nor too few dates. If there are too many dates within the group, what may be a change in energy usage due to a temperature fluctuation would be perceived as a inefficient day. If there are too few dates in a group, then even the smallest of changes in energy output may also be seen as an anomaly. In the next couple of days, I hope to do a bit more research on this topic and also do some testing with code. The Poisson Distribution is a powerful tool, but there are certain guidelines that must be met in order for it to be used. 

Last week was primarily geared towards getting me and Ajay up to speed with what Matt has been doing. We coordinated a couple of Google Hangouts where Matt went over some basic graphing tools in Spyder and a few of the functions that he created. These were very useful and were a nice complement to the other tutorials on working with pandas. We were also able to hold one team meeting at the library where we were finally able to meet face to face. I mainly worked with Matt in trying to fit a log normal distribution to the energy data, which would allow us to visualize where the opportunities are to save money. This week, we are shifting to more independent work with a couple of team meetings here and there. Matt has a list of things that he would like to have accomplished, and I will find something on the list to start working on.

These past few days I have been getting started on multiple aspects of the project. It has been a while since I have been coding, so I started to get back into it by working on some different problems. I went onto the USACO training pages and worked through quite a few of their problems. These problems were very interesting and required using different techniques to solve them. In addition, I have also continued to go through the codecademy tutorials on python since I have been coding in Java for the most part.

Today I worked on getting my work space set up. I downloaded Anaconda from the internet and also got the pandas library installed as well. I had a few problems during the process, but I asked Matt for help and he was very helpful and able to resolve all of the issues that I had. I played around with the Spyder IDE and tomorrow I will work with Ajay on getting a schedule worked out.

School for Andover High students ended last Thursday, so me and Ajay began working on our project today with Anil and Matt. I met with the whole team at the library for a brief meeting before Matt caught me and Ajay up on what he has been doing. Our project will hopefully allow the user to select a file that contains the data usage of the school over a period of time, and then output areas with high potential to save. To begin, Matt gave us an introduction to how to use the pandas library in Python and also described the statistical methods he has been using to calculate the areas of potential. Before we meet again as a team, me and Ajay plan on getting our work setup complete, working more with Python and pandas, and learning more about the statistical methods. This summer's project has a lot of potential and I am very excited to be a part of it.