I've been working on legislative data for 15 years now, on open source scrapers with OpenStates and running a commercial product targeted at professionals (competitor to those in the article).
We tried for years with OpenStates to run a free legislative tracking product before eventually having it partner with a commercial provider who was willing to contribute the resources to keep it alive and help out with the open source pieces (shout out to Plural, nice folks).
Believe me when I say that this space is a classic nerd tar pit. It looks like a relatively easy problem, a few hundred scrapers, search, and some basic CRM functionality and you're off to the races.
The problem is that behind the scenes the data is very complicated, and the sources constantly change and break in goofy ways. You need to be running hundreds of scrapers constantly (many of them against akamai or cloudflare), and working around new source website bugs or procedural edge cases every week. It doesn't scale like something like product or web search where you can just ignore broken pages, the penalty for missing things is too high. Tuning your workflow so people find what they need without getting buried is tough, because there are tens of thousands of bills a session about things people think they care about like "AI" or "taxes". On top of that, the low or zero budget clientele is often that mix of high-expectation and low domain knowledge that makes them a big support burden.
Fiscalnote burned 750 million dollars in VC money on this and just went under this week, granted with a series of spectacular own-goals.
I wish this author the best of luck, and if you want to team up on scrapers please give us a shout. But please be aware that you're promising the moon, and try to build a model that will be financially and effort-sustainable. Keeping this stuff going is a _slog_. I'm really hoping that someone can bring the professional level tools to normal people.
I have some experience in this space and I want to strongly encourage the author to reconsider their free as in beer model.
Yes, your target users don’t have a lot of money, but they also deserve a sense of whether or not you’re going to keep maintaining this project. Additionally, they are generally NOT technical and will not have the skills necessary to set up or maintain this platform.
Without a paid offering, they will have to run the software and will not have any clarity about your long term commitment to the project. Feel free to reach out to me. My email address is in my profile.
The author might like knowing about a similar effort targeting Automated License Plate Readers (ALPR), that was discussed here a few months ago: https://news.ycombinator.com/item?id=46290916.
I appreciate the impetus behind this. But I'm unsure whether this warrants an HN post. The post text is AI-written, and there's no information on technical details--just a kind of vague problem statement. Nor was I able to find any code for the project elsewhere on the site.
There are many good criticisms against data center. And yet, the water issue always comes up first. Must we spew false/untruthhood just so our political message is catchy? I suppose yes - in times of war/politics, the laws/truths are silent. But it doesn't have to be so here.
I've never had it come up first. Neat how 2 people can have 2 opposite experiences based on their different life paths.
Anyways: Since, between our 2 opposite experiences, it might as well be totally random, I don't think the ordering of concerns is that important. Better to focus on substance, like the concerns themselves.
I've been working on legislative data for 15 years now, on open source scrapers with OpenStates and running a commercial product targeted at professionals (competitor to those in the article).
We tried for years with OpenStates to run a free legislative tracking product before eventually having it partner with a commercial provider who was willing to contribute the resources to keep it alive and help out with the open source pieces (shout out to Plural, nice folks).
Believe me when I say that this space is a classic nerd tar pit. It looks like a relatively easy problem, a few hundred scrapers, search, and some basic CRM functionality and you're off to the races.
The problem is that behind the scenes the data is very complicated, and the sources constantly change and break in goofy ways. You need to be running hundreds of scrapers constantly (many of them against akamai or cloudflare), and working around new source website bugs or procedural edge cases every week. It doesn't scale like something like product or web search where you can just ignore broken pages, the penalty for missing things is too high. Tuning your workflow so people find what they need without getting buried is tough, because there are tens of thousands of bills a session about things people think they care about like "AI" or "taxes". On top of that, the low or zero budget clientele is often that mix of high-expectation and low domain knowledge that makes them a big support burden.
Fiscalnote burned 750 million dollars in VC money on this and just went under this week, granted with a series of spectacular own-goals.
I wish this author the best of luck, and if you want to team up on scrapers please give us a shout. But please be aware that you're promising the moon, and try to build a model that will be financially and effort-sustainable. Keeping this stuff going is a _slog_. I'm really hoping that someone can bring the professional level tools to normal people.
I have some experience in this space and I want to strongly encourage the author to reconsider their free as in beer model.
Yes, your target users don’t have a lot of money, but they also deserve a sense of whether or not you’re going to keep maintaining this project. Additionally, they are generally NOT technical and will not have the skills necessary to set up or maintain this platform.
Without a paid offering, they will have to run the software and will not have any clarity about your long term commitment to the project. Feel free to reach out to me. My email address is in my profile.
The author might like knowing about a similar effort targeting Automated License Plate Readers (ALPR), that was discussed here a few months ago: https://news.ycombinator.com/item?id=46290916.
It relies on automated scraping + human confirmation. Louis Rossman describes how it works in https://www.youtube.com/watch?v=W420BOqga_s
I appreciate the impetus behind this. But I'm unsure whether this warrants an HN post. The post text is AI-written, and there's no information on technical details--just a kind of vague problem statement. Nor was I able to find any code for the project elsewhere on the site.
URL doesn't seem to work?
Seems to be changing the .org to a .com
https://firestriker.org/blog/building-firestriker-why-im-mak...
--
related, author is a friend with a less active HN account https://news.ycombinator.com/user?id=blakeofwilliam
Kind of sad to see AI water use as the first listed issue motivating this.
It is a completely fake concern. See here: https://blog.andymasley.com/p/the-ai-water-issue-is-fake
Okay. There are other criticisms of datacenter buildout that make this kind of product valuable. Moving on.
I am more sympathetic to the OG.
There are many good criticisms against data center. And yet, the water issue always comes up first. Must we spew false/untruthhood just so our political message is catchy? I suppose yes - in times of war/politics, the laws/truths are silent. But it doesn't have to be so here.
> the water issue always comes up first
I've never had it come up first. Neat how 2 people can have 2 opposite experiences based on their different life paths.
Anyways: Since, between our 2 opposite experiences, it might as well be totally random, I don't think the ordering of concerns is that important. Better to focus on substance, like the concerns themselves.