These are good practices to keep in mind when setting up GenAI solutions, but I'm not convinced that this part of the job will allow "data scientist" as a profession to thrive. Here's my pessimistic take.
Data scientists were appreciated largely because of their ability to create models that unlock business value. Model creation was a dark magic that you needed strong mathematical skills to perform - or at least that's the image, even if in reality you just slap XGBoost on a problem and call it a day. Data scientists were enablers and value creators.
With GenAI, value creation is apparently done by the LLM provider and whoever in your company calls the API, which could really be any engineering team. Coaxing the right behavior out of the LLM is a bit of black magic in itself, but it's not something that requires deep mathematical knowledge. Knowing how gradients are calculated in a decoder-only transformer doesn't really help you make the LLM follow instructions. In fact, all your business stakeholders are constantly prompting chatbots themselves, so even if you provide some expertise here they will just see you as someone doing the same thing they do when they summarize an email.
So that leaves the part the OP discusses: evaluation and monitoring. These are not sexy tasks and from the point of view of business stakeholders they are not the primary value add. In fact, they are barriers that get in the way of taking the POC someone slapped together in Copilot (it works!) and putting that solution in production. It's not even strictly necessary if you just want to move fast and break things. Appreciation for this kind of work is most present in large risk-averse companies, but even there it can be tricky to convince management that this is a job that needs to be done by a highly paid statistician with a graduate degree.
What's the way forward? Convince management that people with the job title "data scientist" should be allowed to gatekeep building LLM solutions? Maybe I'm overestimating how good the average AI-aware software engineer is at this stuff, but I don't see the professional moat.
I agree with you take the there isn’t a lot of specialist work for data scientists to do with using off-the-shelf LLMs that can’t be done by an engineer. As an AI-aware software engineer myself… this stuff wasn’t that hard to pick up. Even a lot of the work on the Evals side (creating an LLM judge etc.) isn’t that hard and doesn’t require serious ML or stats.
But aren’t there still plenty of opportunities for building ML models beyond LLMs, albeit a bit less sexy now? It’s not like you can run a business process like (say) AirBnB’s search rankings or Uber’s driver marching algorithms on an LLM; you need to build a custom model for that. Or am I missing something here? Or is that point that those opportunities are still there, but the pond has shrunk because so much new work is now LLM-related? I buy that.
> I agree with you take the there isn’t a lot of specialist work for data scientists to do with using off-the-shelf LLMs that can’t be done by an engineer.
Conversely, data scientists are doing software engineering, including webdev. It’s an interesting time. I think it’s less about the job title demarcation now, and more about output.
I agree. It is difficult to convince leadership to do this work at all ("it works on my example, ship it"), and in my experience most DS don't even want to do it.
One of the key value is that it forces some thinking about what is the task you want to solve in the first place. In many cases, it is difficult if not impossible to do it, which implies the underlying product should not be built at all. But nobody wants to hear that.
Doing eval only makes sense if making the product better impacts something the business cares about, which is very difficult to do in practice.
I don't really see why evals are assumed to be exclusively in the domain of data scientists. In my experience SWEs-turned-AI Engineers are much better suited to building agents. Some struggle more than others, but "evals as automated tests" is, imo, so obvious a mental model, and can be so well adapted to by good SWEs, that data scientists have no real role on many "agent" projects.
I'm not saying this is good or bad, just that it's what I'm observing in practice.
For context, I'm a SWE-turned-AI Engineer, so I may be biased :)
I think there's a lot of methodological expertise that goes into collecting good eval data. For example, in many cases you need human labelers with the right expertise, well designed tasks, well defined constructs, and you need to hit interrater agreement targets and troubleshoot when you don't. Good label data is a prerequisite to the stuff that can probably be automated by the AI agent (improving the system to optimize a metric measured against ground truth labels). Data scientists and research scientists are more likely to have this skillset. And it takes time to pick up and learn the nuances.
One thing data scientists brought to the table was statistical rigor in the models, but that seems to have left the building at this point with LLM-based solutions.
I say this quite a lot to data scientists who are now building agents:
1. think of the context data as training data for your requests (the LLM performs in-context learning based on your provided context data)
2. think of evals as test data to evaluate the performance of your agents. Collect them from agent traces and label them manually. If you want to "train" a LLM to act as a judge to label traces, then again, you will need lots of good quality examples (training data) as the LLM-as-a-Judge does in-context learning as well.
I have a data science/engineering background. From my perspective, using AI is like mining the solution space for optimality. The solution space is the combinatorics of the billions of parameters and their cardinalities. You try to narrow down the search space with your prompt and hopefully guide your mining with more semantic-based heuristics towards your optimal solution.
You might hit a local maxima or go down a blind path. I tend to completely start my code base from scratch every week. I would make things more generic, remove unnecessary complexity, or add new features. And hope that can move me past the local maxima.
I can see cases like the recently mentioned pg_textsearch (https://news.ycombinator.com/item?id=47589856) being perfect cases for this kind of development style succeeding - where you have the clear test cases, benchmarks, etc you can meet.
Though for greenfield development, writing the test cases (like the spec) is equally as hard, if not harder than writing the code.
I also observe that LLMs tend to find themselves trapped in local minima. Once the codebase architecture has been solidified, very rarely will it consider larger refactors. In some ways - very similar to overfitting in ML
> The bulk of the work is setting up experiments to test how well the AI generalizes to unseen data, debugging stochastic systems, and designing good metrics.
In my experience, this is missing a big part of the work: confirming what the data actually is, sometimes despite what people think it is.
I just spent yesterday applying Kaparthy's autoresearch on an ML problem.
I teach ML for a living and was amazed with what the tokens gave back to me after many rounds of experiments. If Kaggle was still a thing, AI would generally beat it.
The challenge I've seen is that most data science/ml modeling work is quite weak. Folks don't even know the basic tools well. Not sure if giving AI to them will really open up many doors to them.
As always experts love minions of juniors doing their deeds. Non-experts get to wade through slop.
I agree AI could probably do a decent job on Kaggle problems. Of course, almost no DS job is building models with well-defined objectives and perfect data. The DS and MLE folks I work with mostly spend their time reframing ill-posed product requests into ML systems that can be maintained and improved with feedback loops.
A _huge_ part of a DS is saying "No" to bad ideas posed by non-experts. The issue with LLMs is all they ever say is "Yes" and "Wow, that's such a great idea!"
> The issue with LLMs is all they ever say is "Yes" and "Wow, that's such a great idea!"
I've been obsessed with the idea of self reflecting agents that run an iteration, do some analysis, fix a bug or create an optimization, and run another iteration. I have applied this concept to deterministic problems, such as, create an endpoints API that call websites internal fetch XHR endpoints or create an endpoint that will use browser automation to extract data or fill in and submit a form as a fallback. Because I, the developer, can label all the endpoints to test the agent against, it will keep trying until it finds them all. The other side is take a screen shot of the website and use that as a wire frame for the coding agent to build a UI dashboard. That is deterministic also, it either looks exactly alike and with same behaviors or it doesn't. Then wire the two together. The self reflecting agents can iterate and solve these problems updating their own instructions.
What I describe below has the quality of a slot machine and pays as out nothing as well.
Creating trading backtests have been infinitely more complicated. Can the coding agents find alpha, unsupervised. I've summed up the problem as enforcing two rules; at any moment data that is future of the moment a position, enter or exit, is created can not exist in the system and every model has to be tested on out-of-sample data. I'll work for one day ensuring that no future data enters the system and after the agent will discover the best way to handle null exit values is to skip entering the position. Of course, that happens because options go to $0 so the correct thing would be to enter a position and mark it as a 100% loss.
No matter how hard I try the coding agent will find an experiment with positive results without finding the bug that is causing it and mark the task as complete. Here is the work [0]if you are curious. It is at the point where the project has nothing to do with machine learning or anything else other than hunting bugs. Every time there is a winning strategy, without fault, I will find the look ahead bias that will collapse the model. I haven't found the look ahead bias in some of the models, nonetheless, I know it is there.
I mean it is a similar loop. Define what good looks like, measure how far off you are, iterate. I would say though that the people who've been doing that for years just have a head start that prompt engineers don't.
These are good practices to keep in mind when setting up GenAI solutions, but I'm not convinced that this part of the job will allow "data scientist" as a profession to thrive. Here's my pessimistic take.
Data scientists were appreciated largely because of their ability to create models that unlock business value. Model creation was a dark magic that you needed strong mathematical skills to perform - or at least that's the image, even if in reality you just slap XGBoost on a problem and call it a day. Data scientists were enablers and value creators.
With GenAI, value creation is apparently done by the LLM provider and whoever in your company calls the API, which could really be any engineering team. Coaxing the right behavior out of the LLM is a bit of black magic in itself, but it's not something that requires deep mathematical knowledge. Knowing how gradients are calculated in a decoder-only transformer doesn't really help you make the LLM follow instructions. In fact, all your business stakeholders are constantly prompting chatbots themselves, so even if you provide some expertise here they will just see you as someone doing the same thing they do when they summarize an email.
So that leaves the part the OP discusses: evaluation and monitoring. These are not sexy tasks and from the point of view of business stakeholders they are not the primary value add. In fact, they are barriers that get in the way of taking the POC someone slapped together in Copilot (it works!) and putting that solution in production. It's not even strictly necessary if you just want to move fast and break things. Appreciation for this kind of work is most present in large risk-averse companies, but even there it can be tricky to convince management that this is a job that needs to be done by a highly paid statistician with a graduate degree.
What's the way forward? Convince management that people with the job title "data scientist" should be allowed to gatekeep building LLM solutions? Maybe I'm overestimating how good the average AI-aware software engineer is at this stuff, but I don't see the professional moat.
I agree with you take the there isn’t a lot of specialist work for data scientists to do with using off-the-shelf LLMs that can’t be done by an engineer. As an AI-aware software engineer myself… this stuff wasn’t that hard to pick up. Even a lot of the work on the Evals side (creating an LLM judge etc.) isn’t that hard and doesn’t require serious ML or stats.
But aren’t there still plenty of opportunities for building ML models beyond LLMs, albeit a bit less sexy now? It’s not like you can run a business process like (say) AirBnB’s search rankings or Uber’s driver marching algorithms on an LLM; you need to build a custom model for that. Or am I missing something here? Or is that point that those opportunities are still there, but the pond has shrunk because so much new work is now LLM-related? I buy that.
> I agree with you take the there isn’t a lot of specialist work for data scientists to do with using off-the-shelf LLMs that can’t be done by an engineer.
Conversely, data scientists are doing software engineering, including webdev. It’s an interesting time. I think it’s less about the job title demarcation now, and more about output.
I agree. It is difficult to convince leadership to do this work at all ("it works on my example, ship it"), and in my experience most DS don't even want to do it.
One of the key value is that it forces some thinking about what is the task you want to solve in the first place. In many cases, it is difficult if not impossible to do it, which implies the underlying product should not be built at all. But nobody wants to hear that.
Doing eval only makes sense if making the product better impacts something the business cares about, which is very difficult to do in practice.
I agree with your take.
I don't really see why evals are assumed to be exclusively in the domain of data scientists. In my experience SWEs-turned-AI Engineers are much better suited to building agents. Some struggle more than others, but "evals as automated tests" is, imo, so obvious a mental model, and can be so well adapted to by good SWEs, that data scientists have no real role on many "agent" projects.
I'm not saying this is good or bad, just that it's what I'm observing in practice.
For context, I'm a SWE-turned-AI Engineer, so I may be biased :)
I think there's a lot of methodological expertise that goes into collecting good eval data. For example, in many cases you need human labelers with the right expertise, well designed tasks, well defined constructs, and you need to hit interrater agreement targets and troubleshoot when you don't. Good label data is a prerequisite to the stuff that can probably be automated by the AI agent (improving the system to optimize a metric measured against ground truth labels). Data scientists and research scientists are more likely to have this skillset. And it takes time to pick up and learn the nuances.
One thing data scientists brought to the table was statistical rigor in the models, but that seems to have left the building at this point with LLM-based solutions.
I say this quite a lot to data scientists who are now building agents:
1. think of the context data as training data for your requests (the LLM performs in-context learning based on your provided context data)
2. think of evals as test data to evaluate the performance of your agents. Collect them from agent traces and label them manually. If you want to "train" a LLM to act as a judge to label traces, then again, you will need lots of good quality examples (training data) as the LLM-as-a-Judge does in-context learning as well.
From my book - https://www.amazon.com/Building-Machine-Learning-Systems-Fea...
Yup, agree. “Evaluations” = Tests
Gets pretty meta when you’re evaluating a model which needs to evaluate the output of another agent… gotta pin things down to ground truth somewhere.
I have a data science/engineering background. From my perspective, using AI is like mining the solution space for optimality. The solution space is the combinatorics of the billions of parameters and their cardinalities. You try to narrow down the search space with your prompt and hopefully guide your mining with more semantic-based heuristics towards your optimal solution.
You might hit a local maxima or go down a blind path. I tend to completely start my code base from scratch every week. I would make things more generic, remove unnecessary complexity, or add new features. And hope that can move me past the local maxima.
I can see cases like the recently mentioned pg_textsearch (https://news.ycombinator.com/item?id=47589856) being perfect cases for this kind of development style succeeding - where you have the clear test cases, benchmarks, etc you can meet.
Though for greenfield development, writing the test cases (like the spec) is equally as hard, if not harder than writing the code.
I also observe that LLMs tend to find themselves trapped in local minima. Once the codebase architecture has been solidified, very rarely will it consider larger refactors. In some ways - very similar to overfitting in ML
> The bulk of the work is setting up experiments to test how well the AI generalizes to unseen data, debugging stochastic systems, and designing good metrics.
In my experience, this is missing a big part of the work: confirming what the data actually is, sometimes despite what people think it is.
So true... I get more mileage from just watching an agent work than building sophisticated LLM-as-judge workflows
I just spent yesterday applying Kaparthy's autoresearch on an ML problem.
I teach ML for a living and was amazed with what the tokens gave back to me after many rounds of experiments. If Kaggle was still a thing, AI would generally beat it.
The challenge I've seen is that most data science/ml modeling work is quite weak. Folks don't even know the basic tools well. Not sure if giving AI to them will really open up many doors to them.
As always experts love minions of juniors doing their deeds. Non-experts get to wade through slop.
I agree AI could probably do a decent job on Kaggle problems. Of course, almost no DS job is building models with well-defined objectives and perfect data. The DS and MLE folks I work with mostly spend their time reframing ill-posed product requests into ML systems that can be maintained and improved with feedback loops.
A _huge_ part of a DS is saying "No" to bad ideas posed by non-experts. The issue with LLMs is all they ever say is "Yes" and "Wow, that's such a great idea!"
> The issue with LLMs is all they ever say is "Yes" and "Wow, that's such a great idea!"
I've been obsessed with the idea of self reflecting agents that run an iteration, do some analysis, fix a bug or create an optimization, and run another iteration. I have applied this concept to deterministic problems, such as, create an endpoints API that call websites internal fetch XHR endpoints or create an endpoint that will use browser automation to extract data or fill in and submit a form as a fallback. Because I, the developer, can label all the endpoints to test the agent against, it will keep trying until it finds them all. The other side is take a screen shot of the website and use that as a wire frame for the coding agent to build a UI dashboard. That is deterministic also, it either looks exactly alike and with same behaviors or it doesn't. Then wire the two together. The self reflecting agents can iterate and solve these problems updating their own instructions.
What I describe below has the quality of a slot machine and pays as out nothing as well.
Creating trading backtests have been infinitely more complicated. Can the coding agents find alpha, unsupervised. I've summed up the problem as enforcing two rules; at any moment data that is future of the moment a position, enter or exit, is created can not exist in the system and every model has to be tested on out-of-sample data. I'll work for one day ensuring that no future data enters the system and after the agent will discover the best way to handle null exit values is to skip entering the position. Of course, that happens because options go to $0 so the correct thing would be to enter a position and mark it as a 100% loss.
No matter how hard I try the coding agent will find an experiment with positive results without finding the bug that is causing it and mark the task as complete. Here is the work [0]if you are curious. It is at the point where the project has nothing to do with machine learning or anything else other than hunting bugs. Every time there is a winning strategy, without fault, I will find the look ahead bias that will collapse the model. I haven't found the look ahead bias in some of the models, nonetheless, I know it is there.
[0] https://github.com/adam-s/alphadidactic
I mean it is a similar loop. Define what good looks like, measure how far off you are, iterate. I would say though that the people who've been doing that for years just have a head start that prompt engineers don't.