Major Undertaking--Need A LOT OF Help

[Big W Powah!]

I've been volunteered to develop a VB/Whatever Else is Needed application to pull data from NFL.Com (game stats, etc), parse it to a VB form, and send it to a database on a local machine to be used with our Database Development class.


If anybody is willing to help (this is for no credit), I'd be greatly appreciated. Right now we're in stages of having to pull from the website (without viewing source) to get it to us. This is difficult, as you can imagine.

We need something working roughly by next week, unfortunantly.

If anybody is able and willing, please post ideas/suggestions/code snippets here. Thank you.

[Hachima]

If you just have to pull the data once, all you have to do is copy the table from their site with the headers, paste it into excel. Then arrange the columns to match the columns in your table in MSSQL, if thats what you are using and then paste. You could have all the data populated in 30 min that way.

[Big W Powah!]

Originally Posted by Hachima

If you just have to pull the data once, all you have to do is copy the table from their site with the headers, paste it into excel. Then arrange the columns to match the columns in your table in MSSQL, if thats what you are using and then paste. You could have all the data populated in 30 min that way.

We're using Access, actually.

And my bad on origional post, getting it from ESPN I think. I'll have to check 100%.

We need to pull data, yes, but its more than just tables--we need basically everything ahving to do with the game (game temp, umpire, etc..)

I'll look into that option though.

Just a side note: I know aboslute dick about sports, we were just given print outs of games (which I don't have atm) to do data entry with, to build data for the rest of the class. The teacher said fuck this, and assigned a few people to automate it. I was one of them--I just need to figure this out.

Thanks so much for your help, sir.

[Phelps McManus]

I hope you can use .NET. I haven't tried what you want to do; I am only providing possible starting points of what sounds like an arduous journey.

Do a search on MSDN for "HTML Parsing by ASP.NET XML Web Services" as well as the "HttpGetClientProtocol" and "MatchAttribute" classes (found in the System.Web.Services.Protocols namespace).

If those don't pan out, go up to System.Web namespace and look into HttpRequest.

You can also do Google searches for people who actually used these classes.

Good luck.

[Hachima]

Well it sounds like you will have to manually parse the data out unless there are XML versions of the data already parsed out there on the net, which there often is. I would try to find that first.

[Tannybaker]

Your idea is interesting but it will never work

[Phelps McManus]

Originally Posted by Tannybaker

Your idea is interesting but it will never work

Why wouldn't it work? Looking at the source for NFL scores on ESPN.com, you can run right down to the 'div class="teams"' tag and rip out the team names. Go down 5 lines to 'div class="tscore"' tag and rip off the scores. You can even look for a winner arrow to determine if the game is over.

If you want, you could probably even use XML to record the structure of their divs and parse all kinds of shit.

Be careful how often you run this. If ESPN thinks you are a web crawler bot (which you are), they could ban your IP. I recommend only running manually after games or on a schedule (like Sunday night and Tuesday morning)"

[Big W Powah!]

Originally Posted by Phelps McManus

Why wouldn't it work? Looking at the source for NFL scores on ESPN.com, you can run right down to the 'div class="teams"' tag and rip out the team names. Go down 5 lines to 'div class="tscore"' tag and rip off the scores. You can even look for a winner arrow to determine if the game is over.

If you want, you could probably even use XML to record the structure of their divs and parse all kinds of shit.

Be careful how often you run this. If ESPN thinks you are a web crawler bot (which you are), they could ban your IP. I recommend only running manually after games or on a schedule (like Sunday night and Tuesday morning)"

Good point. We'll just have to hope that doesn't happen, heh. Right now we're looking at archived scores from 2005, and 2006. 2007 will come if and when those first two seasons are done.

Thanks for the help guys. and yes, we have access to full versions of .NET 2003. 6 days left to get a semi-working prototype put together. wish me luck.

[Zippygoose]

While I also know nothing about sports, I'd first try to find a web service out there that you can hook into that pulls this information for you before you go down the rabbit hole of scraping a site.

[Niceshot23]

seeing that this is a development assignment ... im not sure if your teacher would cut you slack on grabbing that data from somewhere other then parsing it yourself .... anyway, the code below is just an eagles eye point of view of how easy the project is ..... the only thing thats difficult about this is that its tideous ... not that its difficult ... oh, and unless im missing something, i dont see a place where the umpire or weather related data is stored on espn so thats not a possibility ... you can only get what you can....



'historical data
private sub StartParse()

dim intYear as integer
dim intWeek as integer
dim iYear as integer
dim iWeek as integer

intYear = 2005
intWeek = 1

for iYear = intYear to 2006
for iWeek = intWeek to 17
call GetGames("http://scores.espn.go.com/nfl/scoreboard?weekNumber=" & intWeek & "&seasonYear=" & intYear & "&seasonType=2"
next iWeek
next iYear

end sub

private sub GetGames(strPage as string)

dim strGameID as string
dim strGameURL as string

dim intStart as long
dim intEnd as long

strGameURL = "http://scores.espn.go.com/nfl/boxscore?gameId="

'on this page, parse for every "http://scores.espn.go.com/nfl/boxscore?gameId=" get the id and put it into strGame ID inside a Loop
strGameID = 250911002

call ParseGame(strGameURL & strgameid)

end sub

Private sub ParseGame(strGameURL as string)

'parse the game .... the webbrowser control gives you the html to the whole page ... thats what you need to parse ... everything on this page follows patterns so parsing it is not hard


end sub

[Big W Powah!]

Originally Posted by Niceshot23

seeing that this is a development assignment ... im not sure if your teacher would cut you slack on grabbing that data from somewhere other then parsing it yourself .... anyway, the code below is just an eagles eye point of view of how easy the project is ..... the only thing thats difficult about this is that its tideous ... not that its difficult ... oh, and unless im missing something, i dont see a place where the umpire or weather related data is stored on espn so thats not a possibility ... you can only get what you can....



'historical data
private sub StartParse()

dim intYear as integer
dim intWeek as integer
dim iYear as integer
dim iWeek as integer

intYear = 2005
intWeek = 1

for iYear = intYear to 2006
for iWeek = intWeek to 17
call GetGames("http://scores.espn.go.com/nfl/scoreboard?weekNumber=" & intWeek & "&seasonYear=" & intYear & "&seasonType=2"
next iWeek
next iYear

end sub

private sub GetGames(strPage as string)

dim strGameID as string
dim strGameURL as string

dim intStart as long
dim intEnd as long

strGameURL = "http://scores.espn.go.com/nfl/boxscore?gameId="

'on this page, parse for every "http://scores.espn.go.com/nfl/boxscore?gameId=" get the id and put it into strGame ID inside a Loop
strGameID = 250911002

call ParseGame(strGameURL & strgameid)

end sub

Private sub ParseGame(strGameURL as string)

'parse the game .... the webbrowser control gives you the html to the whole page ... thats what you need to parse ... everything on this page follows patterns so parsing it is not hard


end sub

Sorry about the lack of updates; thanks guys, this has been helpful as hell. Been working like a busy bee, and by around noon, thanks to you guys, I should have a wokring prototype.

Also: This is a development project for a non-development class--We just need an ability to automate 3 seasons worth of football being entered into an access database, which will then be migrated to a MSSQL server (yes, we HAVE to migrate it, part of the class), which will then be used to be developed into a school-wide fantasy football league (or 3 or 4). This will be developed partially by me, and partially by the rest of the extra-cirricular development crew at our school.


Honestly, I'm doing all this to be able to add "project resolution and management" or some shit like that to my resume. (this is an entirely non-credited assignment. But it needs to get done)

[Sabolin]

If you haven't already figured out a way to pull down the HTML page to parse, I'd just like to recommend cURL as a great command-line tool for downloading web pages to parse. I've used it many times in conjunction w/ VB6 for parsing meteorological data out of national weather service forecast pages automatically.

edit- Ah I see "working prototype" in the post above mine. Oh well, better late than never

[Niceshot23]

Originally Posted by Sabolin

If you haven't already figured out a way to pull down the HTML page to parse, I'd just like to recommend cURL as a great command-line tool for downloading web pages to parse. I've used it many times in conjunction w/ VB6 for parsing meteorological data out of national weather service forecast pages automatically.

edit- Ah I see "working prototype" in the post above mine. Oh well, better late than never

It's not necessary to bring in a 3rd party control when its available to any PC's with internet explorer .... the control i think is called Microsoft Internet Web Browser Control ... or something like that ... I'm sure they found it already or else they'd be way behind.... the command to go to the URL is "webbrowsercontrol1.navigate [expression]" and to get the html its "strHTML = webbrowsercontrol1.html" after you navigate.... its an easy project to undertake ... should only take a few hours to have a stable prototype

[Big W Powah!]

Originally Posted by Niceshot23

It's not necessary to bring in a 3rd party control when its available to any PC's with internet explorer .... the control i think is called Microsoft Internet Web Browser Control ... or something like that ... I'm sure they found it already or else they'd be way behind.... the command to go to the URL is "webbrowsercontrol1.navigate [expression]" and to get the html its "strHTML = webbrowsercontrol1.html" after you navigate.... its an easy project to undertake ... should only take a few hours to have a stable prototype

The project ended up falling apart due to lack of assistance/the change to trying to read .pdf files/the lack of consistancy in the files once transferred to .txt or .xls