Hi there, looking for an alternative to news.google.com that just simply isn’t a Google product. I know it’s not open source per say, but just curious.
Hi there, looking for an alternative to news.google.com that just simply isn’t a Google product. I know it’s not open source per say, but just curious.
I’ve been thinking about using nlp to deal with my feeds.
Are you happy with your solution ? Can you share a bit more about your pipeline?
I am not happy with it yet but that is because I want it to be perfect and it never will be but I do find that I engage with content at a larger scale and more varied than I do when I go to a single source. I am using the nltk features from newspaper for key word extraction + the trending sources to monitor a few hundred sources. Currently I store all the meta data + links ( urls ) + wikipedia links in a pandas dataframe ( which is becoming a problem ) and visualize trends and data about news in a jupyter notebook. For the enhanced summaries + named entity extraction I am using spacy (https://spacy.io/) from there I use SPARQL ( https://en.wikipedia.org/wiki/SPARQL ) to query dbpedia (https://en.wikipedia.org/wiki/DBpedia) to augment entity knowledge ( ex: adding data about the size , industry of a company or summary explanations of scientific concepts, etc ). The named entity matching and augmentation is the portion that needs the most work. Newspaper has some nice caching features so I query all sources everyday but only pull in new articles.
I might play around with moving portions of the data into a graph db and some better ways to query based on concepts. Right now I just write python code to query the pandas DB based on different parameters.
Wow that’s quite developed.
So you consume content in a jupyter notebook? Or you’re interfacing this with a RSS reader?
From what I read the next step is to run it in a real database.
I consume analytics and identify topics I am interested in via jupyter sometimes i just use ipython if I don’t want to leave the terminal – I need to build more of a frontend but I’ve not got there yet. I mostly read the articles in the terminal. And yup my plan is to find a good db but I am not sure what to use yet.
You could probably repackages your upgraded feed into a RSS format that you serve locally. But that can be more hassle than it may worth.
Thanks for the info it encouraged me to try that sometime :)