Romain Kuzniak - Pragmatic AI: Reshaping the Product Opportunity Tree

[00:00:04] Hey everyone. Uh, I'm really happy to be here, uh, being part of this event. I mean, there's a great conference is all over the place. Uh, and it's always a pleasure to share about product and technology. But in the meantime, I've mixed feelings. I've mixed feelings. I have this kind of uh imposter syndrome. Um, I'm gonna talk about AI, but I'm not an AI engineer. I will not talk about uh how we train models because we don't train really models. Um, I'm not gonna talk about a breakthrough algorithm that we put in place that change and um solutionize all problems. I'm not that kind of guy. However, I'm gonna talk about AI. And I have a story to tell.
[00:00:53] It's the story of our relationship with machine learning and AI. And it all started almost if my remote will work. Doesn't work. Yes, it works. Uh, it it started almost 10 years ago. Um, with what we call, uh, the courses drop out. So, I'm Roman, I'm CTO and head of product at Open Classrooms. Uh, you might know, Open Classrooms is an online school. We provide, uh, training programs and pre-courses. And 10 years ago, there was this, uh, trend about learning analytics. So, what is learning analytics? Learning analytics is the analysis of all the traces that a student can let, uh, during his training program. It could be especially on courses, the chapters they saw, the exercises they tried, the success, fail, and it's all this analysis of traces. And, uh, as you know, in data, if you want to be accurate, you need a lot of data. And actually, there's not that much platforms, especially in France, which have, um, millions of data. And we were one of them. We have millions of users and, uh, thousands of courses. So, many people reached out to us, um, just to work on learning analytics. And one of the main use cases in learning analytics is, um, the prediction of success or failure on a course. So, it's what we call course dropout.
[00:02:37] So, at this time, we partnered with, uh, universities, uh, a dozen of universities just to find an algorithm that could predict the success of the failure of a student. I think we spent roughly two years because, you know, the academic time is really different from the, uh, corporate time. And we land on something like pretty, um, pretty efficient actually. Uh, we were pretty happy about this, uh, algorithm. Uh, it was based on machine learning. But you know, there's a difference between, uh, having a proof of concept that works and being ready to put it in production and serve millions of users real time.
[00:03:22] There were expertise we need to hire, uh, there was technical stack we needed to implement. And if I put a really basic, basic, uh, diagram, there was some value in here, but actually, uh, there was an effort that was pretty high.
[00:03:45] But remember, at this time, we were a startup with limited resources. And actually, actually, actually. So, uh, uh, uh, uh.
[00:04:08] There were also, uh, other opportunities, uh, some with the same kind of value, but with a lower effort. Some with like lower value and high effort, so we don't did this. Um, and so, we decided not to move forward. Uh, we decided to pursue opportunities that would bring more value with a lower cost.
[00:04:31] I'd like to.
[00:04:39] So, moving forward, thank you. Um,
[00:04:49] you know, there was big evolution in the, uh, uh, machine learning field, especially in the technical stack. Um, there was new services in the cloud that ease access and ease provide some facilities to train your model and so on. Um, that was great, but, it was not a big change. It actually reduced a bit the cost. But if you think about the opportunity, well, even if we've done previous opportunities, there were still opportunities that were like, uh, more efficient. So, we didn't move forward.
[00:05:30] We didn't move forward until November 2022. Why November 2022? Guess what, uh, the public release of ChatGPT. Sorry, it's interesting. It's not the same kind of AI or machine learning. It's a different value. You don't do prediction with ChatGPT. But it opened new fields, especially in education. I mean, in education, we generally generate a lot of content. So, what we did is that we jumped into, uh, this new technology. Uh, we saw like so many use cases, it could be internally, but also for the customers. And just because we're a school, there were also, uh, an access just to provide skills to our students. But just let's focus on the first one, the value it could create internally and externally.
[00:06:27] It's a different use case as like as I mentioned, sorry. Um, but, the major change for us, and I think for the industry, is the reduction of cost. Thank you.
[00:06:43] Uh, it opened new opportunities with high value, but at a way lower cost. Actually, to use LLMs, you don't need data scientist. And actually, most of them provide APIs, so you don't need a huge stack to use it. So, it reshaped all the opportunity tree. So, what we did is that we started a lot of proof of concept, and we were really fast on having some, uh, great, uh, uh, achievements. No, not yet. Sorry. Um, but there came, uh, other kind of problems. Um, we started to release our first proof of concept, uh, but, you know, LLM is only a part of what you have to give to your customer. You have to build also, um, mechanism for, uh, scalability, quality, uh, security, and so, at every time we wanted to put a new proof of concept in production, we have to rebuild these things.
[00:07:58] So, easy was not easy enough for us, and fast was not fast enough for us. Please.
[00:08:08] So, we came with this idea of providing a service, an internal service that will even help us more to get to production. Faster, easier. We called it AI as a Service, not like, uh, we brainstorm a lot on this. Um, not at all. Um, and what it is? Uh, it's really thinking about, uh, different topics. The first one is really about, um, the experience, uh, not only the experience of the customer, but the experience internally. How could we implement AI the easiest way possible? Um, there's a couple of key points.
[00:08:57] We define first what could be our ideal experience internally. And we thought that our ideal experience was just writing a prompt, write only one line of code to embed this prompt within the platform or the system and deploy. And another parameter was being sure that if we update the prompt, it doesn't need a new deployment. That the first first point. The second point is multi-models. Um, we have to pause here.
[00:09:38] Um, the ecosystem of LLMs is highly volatile. It means that, uh, you just have to see our mailbox. I mean, there's new models almost every week. Uh, with uh, you know, new competitors, new use case, even each model is better than the previous one. So, it's highly volatile. It's hard to say, oh, this model is the right one.
[00:10:04] There's a high volatility also because it's not really a mature technology yet and a mature ecosystem. A couple of examples, for example, I guess you all remember, um, the some at my gates, uh, from a day, from one day to another. Uh, you know, Open AI was close to you, uh, shut down. Um, we've seen that also recently with Gemini and this, uh, image, uh, generation gate. Uh, so they had to remove, uh, from one day to another, uh, these capabilities. So that's a second, uh, point for this high volatility. And the third point is that, um, I do believe and we do believe that, um, in the near future, we will have to handle many models more specialized. Uh, let let me, uh, let me precise that. Uh, you know, in the mid-2000s, there were these, uh, what we call language war, okay? People were just fighting about Java versus .net versus whatever language. But a couple of years later, and right now, it's commonly understood that, um, we have to be polyglots, meaning, uh, we have to use the best language for, uh, the best cases. And I do think that's what will happen with, uh, LLMs, meaning for specific use case, some models will be better than the others. And so, it's how we can provide the best model for a specific use case. So, that's was the, the second point. The third point was about, uh, not having to rebuild, uh, for every, uh, every system the same kind of thing, which are security, sanity checks, the quality management, the cost management.
[00:12:01] And finally, the fourth, uh, one was about being production ready. I mean, not being able to do only proof of concept really fast, but being ready for production really fast.
[00:12:16] So, let's be a bit more technical now.
[00:12:20] So, this is a traditional LLM stack. Uh, you have a prompt catalog where you store your prompt.
[00:12:30] Uh, you have your knowledge base, so it could be a document, it could be a embeddings database, or whatever. There's this LLM service, uh, and you have an orchestrator in the middle of, um, of all of that. just like ensure taking the right prompt, um, you know, ensuring the the knowledge management, calling the LLMs and so on. But also, all the support activities,
[00:12:56] such as quality, cost, security and sanity checks.
[00:13:04] So, what we did, actually, if you can move forward, thanks.
[00:13:10] is that we built a layer on top of that, uh, a really simple API, uh, with only one endpoint. And, uh, we put the prompt catalog on GitHub, so everyone could, um, you know, update and, uh, publish their prompt. And this API, you just need to put your code on the code of the prompt, and the API will do the rest managing all the rest. On top of this API, we built also, uh, a couple of helpers, uh, SDK, um, mainly, uh, especially we started with a JavaScript one. Uh, on top of the JavaScript SDK, we put, uh, a chat, okay? And, um, we built also a Slackbot, and we're currently building PHP and Python, uh, SDK. So, concretely. And if we can move forward. Putting a chatbot in production is as easy as writing a prompt, so it's a YML format. So, this is a mandatory field, you just need the ID, well, name description just to ensure it's not going a mess in our prompt catalog and your prompt. But we also embedded more advanced feature. If you can move forward. You can choose your model. So on the previous one, it's a different one. You can choose also, uh, the type if it's conversational or not.
[00:14:47] Uh, you can, you know, describe where to find your retrievals. Uh, you can put JSON schema if you want to be more strict on your request and, uh, response. You can put optimizers, sanitizers, or manage limits. So it's a really simple YML file. And, um, oh, yeah, I have to say also the prompt could take some parameters, so in your API or root, you can send parameters on the request and it could, um, fill your your prompt. So, you write your prompt, and then, if you can move forward,
[00:15:25] we created a generic chat. So, you just have to put, and you can see up, I think it's, uh, yeah, I forgot that line. Yeah, prompt code. So, you put the ID of the prompt. You can pass parameters, so it's not mandatory. Um, and then, in production, right away, if you deploy, you have a prompt and a chatbot. So, that's what we did, because we have so many use cases and we didn't want to replicate again and again the same thing. Um, another example is our Slackbot. If you can share the next screen.
[00:16:06] So, basic, basic things, and this chatbot you could also, uh, put your, uh, prompt in a catalog and make it available in a chatbot. So, what does it change?
[00:16:18] Well, the cost of implementation was already low, but it's already null now. So, it's a big change. From this orange dot where we were on the, um, learning analytics, we went to almost no efforts, but lot of value, um, capabilities or opportunities.
[00:16:45] So, I could have stopped my presentation here and say, ah, look at how it's beautiful, how we're super. Uh, well, the reality is, it's not that easy. And we're facing new challenges, and I'm gonna talk about these. The first two ones are really about, um, the value. Uh, and what do I mean? Next one.
[00:17:13] I guess you've all seen all these, uh, promotions in your mailbox, everywhere, every tool, every single tool just provide AI capabilities.
[00:17:23] But who among you already uses it? I will give you a personal experience that that I faced, uh, this week. I was using a diagram tool, and I was just putting like three square, you know, just a a quick diagram with one word per square. So, I selected my three squares, just, you know, to to make the alignment automatically. And then a pop-up showed up and said, oh, do you want a summary of your three squares? And I I'm like, what? I mean, here, is there any kind of value just to summarize three squares? So that that's what I call the AI temptation. It's not because it's easy that we have to put it everywhere. We have to think about what is the value we want to provide. Um, the AI temptation is also linked to the ecosystem. I think for every company and every investor, it's important to show that, oh, we're using AI.
[00:18:24] But we should always think about what what is the value we want to provide to the users. Um, and most of the time, the the real value and the real value of AI is not visible. So, the first challenge is is about not falling into the AI temptation.
[00:18:41] The other challenge is still on value is at the other side of the spectrum. It's the missed opportunities.
[00:18:51] Because for years, we trained our brain not to think about some opportunities that could be like, uh, really costly, our brain didn't always, doesn't always think about, uh, those opportunities that could be now accessible through AI. So, it's how we reshape the way we think and put, okay, but is it possible now with AI? And most of the time, it opens like really new opportunities. I got a couple of example within my teams. Um, I will take the example of this designer that, you know, we're doing orientations. So, we have helpers for the students to find the right training program, but it's quite simple. And we always think thought about, okay, how could we put something more interactive and so on.
[00:19:44] And, you know, with LLM, it's way much easier.
[00:19:49] But we have to reprogram ourselves, and there's still this designer that always come back, oh, it's been a while since I'm talking about having a wizard that could well, could help the students, but, you know, with this kind of chat and this kind of LLMs, it's easy to to do.
[00:20:07] But we have to rethink the way, uh, we see things. So that's for value.
[00:20:14] The third one is about the information. And, um, I have to be honest with myself, and I guess most of the company has to be honest with themselves, the knowledge management has always been overlooked. We all think it's important. We all say, oh, knowledge management, managing the documentation is important, but we didn't put, oh, actually, I didn't put enough effort on this.
[00:20:42] Why? Because if your documentation is not up-to-date, that's okay, your production will not be broken. Um, but it changed with LLMs, and that's the limits we have.
[00:20:57] To have an efficient LLMs, you need a strong also information architecture and knowledge management. And it becomes production, meaning, I guess you've seen this, uh, I think it was last week or two weeks ago, this thing where a chatbot just said, uh, wrong things to the customer. I think it was, uh, Canada, uh, Canada Airline or something. And so, provide discount that was not accurate, and they had to pay for that. Um, so now the information becomes production topic. So, it has to be maintained, updated, always up to date. You have to check, you have to to ensure you have the good patterns and so on. So here it's a new challenge.
[00:21:41] The fourth one is about, um, the new activities. management and it becomes production. Meaning I guess you've seen this, uh, I think it was last week or two weeks ago, this thing where a chatbot just said, uh, wrong things to the customer. I think it was a Canada Canada line or something. And so it provide discounts that was not accurate and they had to pay for that. Um, so now the information becomes production topic. So it has to be maintained, updated, always up to date. You have to check, you have to to ensure you have the good patterns and so on. So here it's a new challenge.
[00:21:41] The fourth one is about, um, the new activities.
[00:21:48] There's new activities and among them I can, uh, I can cut the, um, the prompt engineering. It's easy to do a small prompt that you're using on your web browser. Now when you do it in production for many users, prompts are becoming more and more complex. You have to predict the results. And, uh, it's not that easy. I mean, that's new skills that are coming. Um, and new skills, new jobs, so that's a new activity. Uh, and we're not necessarily equipped. Um, the second one is about, uh, quality. How do you ensure quality, uh, in the LM world? So, for most of the time, quality could be automated, but, you know, here the results could differ from one request to another. Plus, how do you ensure, um, the relevancy of this, uh, answer? So, potentially new stack, potentially new way of thinking the quality.
[00:22:54] And finally, I talked about it previously, um, you know, the management of the information. We we have knowledge managers sometimes in companies, but it's different to manage a knowledge internally than nurturing a prompt. And the final challenge we have is about, um, the organization.
[00:23:19] Uh, with the rise of LLM's, we talk about, we talk a lot about, uh, the impact on individuals. How LLM's will impact the jobs, will it remove some jobs, will it change how people should engage with that just for their future. Uh, but we don't talk that much about organizations. And, uh, the classical cross-functional team that we can call squad with a product manager, a designer, engineers. Uh, sometimes you can put data analysts, quality, well. This one might change especially because. With service like the one we built for this kind of use case, you don't need engineers. Uh, it doesn't mean that we have to remove engineers, but. You know, for example, this uh chat that we built to help uh the students, we don't need engineer to to put in place. Not to maintain. So it's rethinking how we are organized, uh, in this kind of cross-functional team, but also among the all organization. So those are the the challenges.
[00:24:36] To to wrap it up.
[00:24:39] Uh, of of course, and obviously, uh, LLM's brings, uh, new use cases that really, uh, provide new opportunities, uh, for companies and customers. But one of the biggest impact of the rise of LLM's is the reduction of the cost and the accessibility of AI to any kind of company without data scientist, for example.
[00:25:03] And in the meantime, it raised new challenges, new activities that we have to face, that we have to, uh, uh, go through. But I will conclude that there's a lot of uncertainties right now, there's a lot also of opportunities. And I do really think we we're living, uh, an exciting time. That's all for me. Thank you for having me and I'm really happy to answer any of your questions. Thank you.
[00:25:44] Je vais bientôt vous laisser.
[00:25:48] You're on that.
[00:25:51] I think we can have question in English and also a few in French if you were able to.
[00:25:57] Hey, thank you. You mentioned you could automate quality, could you elaborate a little bit on that? Uh, what are the main approaches? Thank you.
[00:26:07] I know, um, I think, I know, I might not have been clear, um, the quality evolution was really about, uh, making automatic. One of the challenges with the quality with LLM's is how could you make it automatic, so we've not succeeded, it's a challenge. But I think automation is not enough. Uh, there are so many use cases where you have to ensure with LLM's is the relevancy of the answer. And how you deal with that at scale. I mean, there's not that much white paper on that yet, so this is a a strong challenge. Sorry, I don't have the answer for the automation, I would have. Okay.
[00:26:55] Hello. Uh, thank you for the talk. Um, how do you deal with the cost of it? Uh, how do you manage that? Yeah. Thank you.
[00:27:08] So, um, well, there's many ways to to to manage the cost. So, uh, it's through, well, first through the infrastructure services that can give you the cost. Now it's about rate limiting also that we show. So it's not only rate limiting, it's also token limits like the the size of the content. So if a student just copy paste a whole book, you know, it could be really costly, so that's why we manage that. But what we saw right now, it's not really at big scale, we're not talking about millions of people. It's also like the kind of feature you provide at some point. Because, um, here we're talking about students. Uh, if we were serving AI for millions of visitors, it would have been different, obviously. Uh, but today, I mean, we were really surprised that cost is not really an issue. Um, I will give you an example, we use AI also internally, so uh, you know, the Slackbot and other things. But when you go through the API, the costs are quite really low. And, uh, I think last months and we are quite a pretty extensive usage, I think we're under 200, uh, euros per month. And if you think about, uh, the number of people that use it and you take the, um, license of a chat GPT. I mean, the ratio is like incredible. So, um, for now, course is not, uh, it's not an issue yet. But because we choose also our, um, use cases. If we put AI on for all the, uh, millions of visitors we we have, yeah, might be, uh, might be a challenge. Thank you.
[00:29:01] Uh, hello. You talked about, uh, the AI in general, but, um, did you make the difference between the predictive one and the generative one?
[00:29:11] Yes, we did and, uh, I think that's all the purpose. We were not equipped in terms of skills to manage, uh, you know, machine learning and prediction and recommendations. And the big change is that to use this AI generative one, you don't need data scientists. I do believe you don't need, or if you want really to to do a customized model potentially. But for most of the use case we have, we don't need. And I think that's a big difference. Because, um, I don't think the previous AI will disappear, I mean, there's really specific use cases for for each one. But the big change is the accessibility to AI for for us, don't need experts, at least for now.
[00:30:05] Hello. You talked about also the values or the, uh, the skills to have, to to have the quality, uh, about AI. Do you think that we have enough, uh, people that have the old skills today?
[00:30:22] It it it's definitely new, uh,
[00:30:25] new skills. Uh, and I think it's also a new field. I mean, prompt engineering, you know, in specialized blog you see that it's evolving really fast.
[00:30:35] It's evolving also with, uh, with the models. Um, you know, potentially we're going to see, you know, this classic job postings, I need a prompt engineer with 10 years of experience. But, uh, uhm, well. Right now, I think, uh, I don't even think there are some training, maybe maybe in the US a couple of ones. Um, so here, here it's a challenge. How do you find these people? And how do you find also the right practices, we're talking about, we're talking about quality. There's some white papers, but it's not like. You know, it's always evolving.
[00:31:13] So, yeah. But, you know, it's interesting what you said because. Uh, one of the axis of open classrooms is also what we we provide training programs. Uh, but we embodied the fact that LLM's will change definitely, um, the the market and the jobs. And so in every single training program, on the first courses, it's about AI and, um, generative AI. So we train our students just to, uh, ensure, uh, they they will use it. Well, it's not like prompt engineer, uh, training program, but I think, yeah, it will take time at some point, my opinion.
[00:32:02] On on va en prendre une autre si éventuellement en français vous vous sentez plus à l'aise, vous pouvez, hein, je répète.
[00:32:08] Thank you for the presentation. Euh
[00:32:12] Um, you haven't talked about matrix. Have you seen a difference?
[00:32:18] NPS, time to market, I don't know.
[00:32:22] Uh, uh, no, I think. What is your question exactly? I haven't talked about matrix.
[00:32:30] But you see that to do AI implementation. Have you seen the difference in metrics?
[00:32:37] Oh, okay. Well, it it it depends really on use cases. So, um, I can take one where we're more advanced, it's about uh the support to the students. Uh, and um, obviously here it's uh, we're measuring if I would be a bit technical on uh on education, but we're measuring it's the uh speed of patience to acquire or the speed of skill acquisition.
[00:33:07] And when we put the that in place first in a couple of training program, uh, we see quite quite a significant increase. It's not only AI, it's not like a magical tool. If you don't have the right content, if you don't have the right other kind of support. Um, so what was quite difficult here is to really identify what is really because of this, uh, AI support versus other parameters. But yeah, in a product driven approach, there's always like, okay, what do you want to move when you're doing an initiative, so it really depends on on the use cases. We put like uh targets and uh key results that we monitor. Um, I don't know if it answer uh your your question, sorry, it was other kind of metrics you were talking about.
[00:34:00] Euh It was really the impact on the metrics.
[00:34:05] Attends, juste le micro.
[00:34:11] Yeah. Uh, yes, kind of, but, um, have you seen a difference? What would be the stake if you haven't implemented AI?
[00:34:24] What would have been the
[00:34:25] Bon, je vais le faire en français. Si vous aviez pas implémenté Lia sur certains use case.
[00:34:30] Ouais.
[00:34:31] Quels auraient été les les les enjeux en quoi ça vous a fait gagner de la croissance ou bien? Ça, c'est important.
[00:34:38] Euh, alors on mesure pas mal de choses chez nous, notamment en interne. Je pense que euh là on a j'ai parlé beaucoup de de cas d'utilisation externe à destination de nos utilisateurs mais l'essentiel de de l'AI chez nous elle est plutôt interne. Euh, dans différents secteurs et je vais prendre un peu celui de de de l'éducation sur la formation donc à chaque fois on mesure le temps qu'on met pour euh créer un cours,
[00:35:04] créer des formations, ça peut être plus ou moins long, un cours ça met à peu près un mois, un mois et demi, mais une formation longue ça peut mettre 6, 9 mois. Euh, j'ai pas le chiffre exact mais grosso modo, je crois que c'est 25 à 30 % de de diminution de de temps du cycle time là-dessus. Ce qui est euh, ce qui est euh énorme quand on parle de d'un parcours de 6 à 9 mois, de passer de 9 à 6, c'est conséquent. Euh, c'est aussi sur la qualité aussi parfois. Euh, et puis c'est aussi sur après il y a il y a d'autres mécaniques de de productivité mais après c'est difficile de sortir exactement est-ce que il y a que l'IA parce qu'en fait on a plein d'initiatives. Euh, nous on mesure par exemple sur sur l'ingénierie, on mesure ce qu'on appelle le cycle time, donc le temps entre euh je commence une tâche et aller en production. On l'a beaucoup diminué euh euh ces six derniers mois. C'est pas juste l'AI, hein, un petit bout de l'AI, euh donc difficile ici de de de sortir. Après, je pense qu'il y a aussi une autre mesure, c'est l'expérience utilisateur ou l'expérience employé. C'est qu'en fait ce qui est facile à automatiser avec l'AI, c'est aussi les tâches à faible valeur d'une certaine façon. On l'utilise pas mal en user research par exemple, on a beaucoup de de feedback et donc mettre la taxonomie sur les feedback. Euh, c'était pas forcément simple auparavant, là on a de très bons résultats, euh, quand tu dois gérer 5000 feedback à la main, la taxonomie. Euh, donc là aussi c'est c'est des mesures, en plus ça peut ne c'est pas ça qui a le plus de valeur, c'est après l'analyse qu'on fait. Donc euh à l'échelle de l'entreprise, c'est difficile de dire mais sur certains cas spécifiques, on voit des changements drastiques. Merci.
[00:36:58] Just a quick question. Thank you for your presentation. What you presented, the API and some part of the system, is it open source or accessible or did you write a blog post about it?
[00:37:12] Um, now I was expecting this question. Um, well, we tried in the past to open source some of our libraries, but, um, as you know, open source is require a lot of time. Uh, just for for the story, it was really, the first one was really a small library to to help, uh, for a translation tool, it was a client. But already for a small library like this, it required like a lot of work on our side. Right now we're a bit stretch in terms of resources. I think we're going to publish more details. Um, to be honest, that's not that much work, uh, would be a lot of work to maintain actually because, uh, we decided for now to to be really specific on some models and so on. Uh, but no, unfortunately, we we don't plan to to make it open source. I think we're going to publish just to to explain a bit more. But if you want after after the conference we we can talk if you need more information.
[00:38:24] D'autres questions?
[00:38:27] Merci. Thank you very much.

Transcript