<html>
<p>I've seen many articles online describing how the poisson distribution could potentially be used as a means of predicting soccer scores. However, I haven't seen much in the way of practical examples.</p>
<p>It seems to be me that this might be an ideal subject for my first post on Steemit.</p>
<p>Now, before you read any further, I'm a hobbyist, so don't expect high quality code! I've just recently started learning Python so this will be a learning process for me more than anything. I taught myself 6502 assembly language in my early teens and wrote a couple of (unpublished) games on the Commodore 64. I was also involved in the demo scene and coded many a fine starfield. Later, I moved the Amiga and 68000 code and some C, before everything went PC and I ended up using Visual Basic. </p>
<p>I drifted away from programming for a long time but now that I'm using Linux, I've got the bug again, so I've been learning Python. Big shout out to <a href="pythonprogramming.net">pythonprogramming.net</a> & <a href="www.youtube.com/user/sentdex">www.youtube.com/user/sentdex</a> for getting me this far.</p>
<p>But enough about me, lets get to work.</p>
<p>First thing we need is some data, so we're going to use Beautiful Soup to scrape some from the web.</p>
<p>There are plenty of places to find historical soccer results, some with ready made .CSV files that can be downloaded, but I've decided to scrape raw data from the Soccer Punter website (<a href="https://www.soccerpunter.com">www.soccerpunter.com</a>). There are a couple of reasons for this choice but the main one is that the .CSVs available usually seem to take a few days before they're updated with the latest results.</p>
<p>I'm going to assume you all know how to install pandas, beautifulsoup and selenium or are clever enough to find out how elsewhere. ;-)</p>
<p>Here's what I've came up with so far...</p>
<blockquote><strong>import </strong>pandas <strong>as </strong>pd<br>
<strong>from </strong>bs4 <strong>import </strong>BeautifulSoup <strong>as </strong>bs<br>
<strong>from </strong>selenium <strong>import </strong>webdriver<br>
<strong>import </strong>datetime<br>
<br>
<strong>def </strong>scrapeseason(country, comp, season):<br>
<em># output what the function is attempting to do.</em><br>
<em> </em>print(<strong>"Scraping:"</strong>, country, comp, str(season)+<strong>"-"</strong>+str(season+1))<br>
baseurl = <strong>"http://www.soccerpunter.com/soccer-statistics/"</strong><br>
<strong> </strong>scrapeaddress = (baseurl + country + <strong>"/" </strong>+ comp.replace(<strong>" "</strong>, <strong>"-"</strong>).replace(<strong>"/"</strong>, <strong>"-"</strong>) + <strong>"-"</strong><br>
<strong> </strong>+ str(season) + <strong>"-" </strong>+ str(season + 1) + <strong>"/results"</strong>)<br>
print(<strong>"URL:"</strong>, scrapeaddress)<br>
print(<strong>""</strong>)<br>
<br>
<em># scrape the page and create beautifulsoup object</em><br>
<em> </em>sess = webdriver.PhantomJS()<br>
sess.get(scrapeaddress)<br>
page = bs(sess.page_source, <strong>"lxml"</strong>)<br>
<br>
<em># find the main data table within the page source</em><br>
<em> </em>maintable = page.find(<strong>"table"</strong>, <strong>"competitionRanking"</strong>)<br>
<br>
<em># seperate the data table into rows</em><br>
<em> </em>games = maintable.find_all(<strong>"tr"</strong>)<br>
<br>
<em># create an empty pandas dataframe to store our data</em><br>
<em> </em>df = pd.DataFrame(columns=[<strong>"date"</strong>, <strong>"homeTeam"</strong>, <strong>"homeScore"</strong>, <strong>"awayScore"</strong>, <strong>"awayTeam"</strong>])<br>
<br>
idx = 0<br>
today = datetime.date.today()<br>
<br>
<strong>for </strong>game <strong>in </strong>games:<br>
<br>
<em># these lines filter out any rows not containing game data, some competitions contain extra info.</em><br>
<em> </em><strong>try</strong>:<br>
cls = game[<strong>"class"</strong>]<br>
<strong>except</strong>:<br>
cls = <strong>"none"</strong><br>
<strong> if </strong>(<strong>"titleSpace" not in </strong>cls <strong>and "compHeading" not in </strong>cls <strong>and</strong><br>
<strong> "matchEvents" not in </strong>cls <strong>and "compSubTitle" not in </strong>cls <strong>and </strong>cls != <strong>"none"</strong>):<br>
<br>
datestr = game.find(<strong>"a"</strong>).text<br>
gamedate = datetime.datetime.strptime(datestr, <strong>"%d/%m/%Y"</strong>).date()<br>
<br>
<em># filter out "extra time", "penalty shootout" and "neutral ground" markers</em><br>
<em> </em>hometeam = game.find(<strong>"td"</strong>, <strong>"teamHome"</strong>).text<br>
hometeam = hometeam.replace(<strong>"[ET]"</strong>, <strong>""</strong>).replace(<strong>"[PS]"</strong>, <strong>""</strong>).replace(<strong>"[N]"</strong>, <strong>""</strong>).strip()<br>
awayteam = game.find(<strong>"td"</strong>, <strong>"teamAway"</strong>).text<br>
awayteam = awayteam.replace(<strong>"[ET]"</strong>, <strong>""</strong>).replace(<strong>"[PS]"</strong>, <strong>""</strong>).replace(<strong>"[N]"</strong>, <strong>""</strong>).strip()<br>
<br>
<em># if game was played before today, try and get the score</em><br>
<em> </em><strong>if </strong>gamedate < today:<br>
scorestr = game.find(<strong>"td"</strong>, <strong>"score"</strong>).text<br>
<br>
<em># if the string holding the scores doesn't contain " - " then it hasn't yet been updated</em><br>
<em> </em><strong>if " - " in </strong>scorestr:<br>
homescore, awayscore = scorestr.split(<strong>" - "</strong>)<br>
<br>
<em># make sure the game wasn't cancelled postponed or suspended</em><br>
<em> </em><strong>if </strong>homescore != <strong>"C" and </strong>homescore != <strong>"P" and </strong>homescore != <strong>"S"</strong>:<br>
<em># store game in dataframe</em><br>
<em> </em>df.loc[idx] = {<strong>"date"</strong>: gamedate.strftime(<strong>"%Y-%m-%d"</strong>),<br>
<strong>"homeTeam"</strong>: hometeam,<br>
<strong>"homeScore"</strong>: int(homescore),<br>
<strong>"awayScore"</strong>: int(awayscore),<br>
<strong>"awayTeam"</strong>: awayteam}<br>
<em># update our index</em><br>
<em> </em>idx += 1<br>
<strong>else</strong>:<br>
<em># it's a future game, so store it with scores of -1</em><br>
<em> </em>df.loc[idx] = {<strong>"date"</strong>: gamedate.strftime(<strong>"%Y-%m-%d"</strong>),<br>
<strong>"homeTeam"</strong>: hometeam,<br>
<strong>"homeScore"</strong>: -1,<br>
<strong>"awayScore"</strong>: -1,<br>
<strong>"awayTeam"</strong>: awayteam}<br>
idx += 1<br>
<br>
<em># sort our dataframe by date</em><br>
<em> </em>df.sort_values([<strong>'date'</strong>, <strong>'homeTeam'</strong>], ascending=[<strong>True</strong>, <strong>True</strong>], inplace=<strong>True</strong>)<br>
df.reset_index(inplace=<strong>True</strong>, drop=<strong>True</strong>)<br>
<em># add a column containing the season, it'll come in handy later.</em><br>
<em> </em>df[<strong>"season"</strong>] = season<br>
<strong>return </strong>df<br>
<br>
<em># set which country and competition we want to use</em><br>
<em># others to try, "Scotland" & "Premiership" or "Europe" & "UEFA Champions League"</em><br>
country = <strong>"England"</strong><br>
competition = <strong>"Premier League"</strong><br>
lastseason = 2016<br>
thisseason = 2017<br>
<br>
lastseasondata = scrapeseason(country, competition, lastseason)<br>
thisseasondata = scrapeseason(country, competition, thisseason)<br>
<br>
<em># combine our data to one frame</em><br>
data = pd.concat([lastseasondata, thisseasondata])<br>
data.reset_index(inplace=<strong>True</strong>, drop=<strong>True</strong>)<br>
<br>
<em># save to file so we don't need to scrape multiple times</em><br>
data.to_csv(<strong>"data.csv"</strong>)<br>
</blockquote>
<p>Okay, that's enough for now. If you run this you'll have a file called data.csv. Load it up in a spreadsheet and confirm it looks OK and I'll be back soon with some code to do something with our new data.</p>
<p>In the meantime, If anyone has any questions, tips, advice or abuse they'd like to share, please do.</p>
<p><br></p>
</html>
hiveblocks